A burst plug for positioning in a downhole fluid passage has: a shear disc installable in a sealing position in the fluid passage. The shear disc is openable by applying fluid at a burst pressure against the shear disc. The burst plug has a retainer and sleeve formed to retain the shear disc in a fluid flow blocking position after the shear disc is removed from its sealing position. The sleeve has a shear disc retaining position in locking engagement with the retainer when fluid at burst pressure is in communication therewith. The sleeve thereafter assumes a shear disc removal position when fluid in communication therewith is reduced to a selected pressure lower than the burst pressure. In the shear disc removal position, the sleeve and retainer disengage from the blocking position, allowing the shear disc and retainer to be expelled from the fluid passage.