In a non-volatile memory system, a programming operation applies programming pulses to a target word line, determines when a specified number of the non-volatile storage elements reach a defined verify level, and counts a specified number of the programming pulses after the specified number of the non-volatile storage elements reach the defined verify level. Upon completion of the counting, faster-programming storage elements are distinguished from slower-programming storage elements. Programming continues for of at least some of the faster-programming non-volatile storage elements, with an associated programming speed-based slow down measure imposed thereon, and for at least some of the slower-programming non-volatile storage elements without imposing a programming speed-based slow down measure.