A removable hydraulic set packer has a piston activated by fluid in the bore of the packer's mandrel. The piston disengages from the mandrel and pushes a lower wedge unit towards an upper wedge unit. The wedge units push a slip outward from the mandrel toward the surrounding casing. The packer uses a delay between the setting of the slip and the compression of a packing element. In this way, the piston's movement does not compress the packing element until the slip has been able to fully engage the casing. Once the slip is set, the piston assembly can then move a lower gage ring and compresses the packing element against a fixed gage ring fixed. The delay can use a temporary connection between the piston and the lower wedge unit and can use a shoulder distanced from the lower gage ring. Alternatively, a secondary piston with dogs can provide the delay.