A programming techniques adaptively sets a pass voltage and an initial program voltage based on a programming speed of a set of memory cells. In one pass of a multi-pass programming operation, a programming speed-indicating program voltage is obtained. For example, this can be a final program voltage or a program voltage at another programming milestone. A pass voltage is determined for another programming pass of the multi-pass programming operation, by providing an adjustment to a reference pass voltage. An initial program voltage is determined for the another programming pass based on an offset from the programming speed-indicating program voltage. The initial program voltage is further adjusted to counteract an effect of the adjustment to a reference pass voltage. The adjustment to the initial program voltage is opposite in polarity and smaller in magnitude than the adjustment to the reference pass voltage.