The disclosure is directed to techniques for evaluating temporal quality of encoded video. Instead of estimating jerkiness based solely on frame rate or motion activity, the number of consecutive dropped frames forms a basic estimation unit. Several human visual system factors, such as sensitivity to temporal quality fluctuation and motion activity, have been taken into account to make the predicted jerkiness more consistent with the actual human visual response. The temporal quality metric can be used to estimate human perceived discomfort that is introduced by temporal discontinuity under various combinations of video shots, motion activity and local quality fluctuations. The techniques can be applied in two modes: (1) bitstream or (2) pixel mode. The quality metric can be used to evaluate temporal quality, or to control encoding or decoding characteristics to enhance temporal quality.