Techniques and tools are described for scalable video coding and decoding. For example, a video encoder and decoder perform adaptive update operations in lifting-based motion-compensated temporal filtering and inverse motion-compensated temporal filtering, respectively. The encoder, in a prediction stage, computes a motion-compensated prediction for an odd image and computes a high-pass image as the difference between the odd image and the prediction. For an update stage, the encoder performs motion compensation on the high-pass image (and potentially another high-pass image) to compute a motion-compensated update. Based upon perceptual criteria, the encoder adaptively combines the update with an even image to form a low-pass image. The decoder, for an update stage, performs motion compensation on one or more high-pass images to compute a motion-compensated update. Based upon perceptual criteria, the encoder adaptively combines the update with a low-pass image to reconstruct an even frame.