An retaining device of an electrical connector assembly for use with a heat dissipating device, includes a first retainer, a fixing lever, and a second retainer. The fixing lever has one end pivotally mounted to the first retainer and another end capable of locked by the first retainer. The second retainer is integral formed with an elastic plate. The first retainer and the second retainer are two separated members, and the fixing lever and the elastic plate are used for pressing two opposite ends of the heat dissipating device.