This disclosure provides a clock recovery circuit having a phase-locked loop (PLL) with two-point modulation. A binary phase-error signal controls a variable frequency oscillator's (VFO's) feedback path, while a linear phase-error signal controls the PLL outside of that feedback path. The linear phase-error signal is injected using an ultra-low latency delay path. While the binary phase-error signal sets the lock-point of the PLL, the linear phase-error path dominates at high frequencies and also helps reduce dither jitter. Other optional features include an area-efficient hybrid phase detector that generates both the binary and linear phase-error signals, use of a phase interpolator inside the PLL to further smooth dither jitter, recovered clock update filtering for specific data transitions, and support for multi-PAM signaling.