An endpoint or other communication device of a communication system includes a clock recovery loop having a phase error estimator. The communication device is operative as a slave device relative to another communication device that is operative as a master device. The clock recovery loop is configured to control a slave clock of the slave device responsive to a phase error estimate generated by the phase error estimator so as to synchronize the slave clock with a master clock of the master device. The phase error estimator comprises a plurality of filters each configured to generate a different estimate of master clock phase using at least a subset of a plurality of packets received from the master device, and control logic for adaptively selecting at least a particular one of the plurality of filters for use in generating the phase error estimate to be processed in the clock recovery loop.