An electrical submersible pump (“ESP”) has a seal member that carries a packer. A retainer initially retains the packer in a fixed axial position with the seal member as the ESP is lowered into a well conduit. The retainer is releasable after the packer has been set in the conduit in response to a thermal growth axial force on the seal member, enabling relative axial movement between the seal member and the packer. The seal member has annular seal rings that extend over an axial length on the tubular member that is greater than an axial length of the bore of the packer.