Techniques are provided for periodically monitoring and adjusting the threshold voltage levels of memory cells in a charge-trapping memory device. When a criterion is met, such as based on the passage of a specified time period, the memory cells are read to classify them into different subsets according to an amount of downshift in threshold voltage (Vth). Two or more subsets can be used per data state. A subset can also comprise cells which are corrected using Error Correction Code (ECC) decoding. The subsets of memory cells are refresh programmed, without being erased, in which a Vth upshift is provided in proportion to the Vth downshift. The refresh programming can use a fixed or adaptive number of program pulses per subset. Some cells will have no detectable Vth downshift or a minor amount of Vth downshift which can be ignored. These cells need not be refresh programmed.