In a PAM-N receiver, sampler reference levels, DC offset and AFE gain may be jointly adapted to achieve optimal or near-optimal boundaries for the symbol decisions of the PAM-N signal. For reference level adaptation, the hamming distances between two consecutive data samples and their in-between edge sample are evaluated. Reference levels for symbol decisions are adjusted accordingly such that on a data transition, an edge sample has on average, equal hamming distance to its adjacent data samples. DC offset may be compensated to ensure detectable data transitions for reference level adaptation. AFE gains may be jointly adapted with sampler reference levels such that the difference between a reference level and a pre-determined target voltage is minimized.