A plug for a seal bore in a packer mandrel has a shiftable annular member that can selectively open bypass ports to facilitate latching and then be shifted as part of a release from the plug by a running tool to close the bypass passage that go around a frangible barrier that will later be broken by impact force. The annular member has minimal structure internally to allow attachment of the running tool. The annular member drillout proceeds quickly with minimal cuttings and the frangible member is broken by impact. On an assembly with multiple packers getting plugs a trip is saved as a plug is delivered into a lower packer with a string supporting the packer above. The plug is set in the lower packer allowing release of the running string for subsequent placement and setting of the next packer in the same trip.