To estimate content complexity of a video, energy of prediction residuals is calculated. The prediction residuals are usually smaller when the video is less complex and more predictable. Scales of prediction residuals also depend on encoding configurations, for example, I pictures usually have larger prediction residuals than P and B pictures even when the contents are very similar and thus have similar perceived content complexity. To more closely reflect the content complexity, alignment scaling factors are estimated for different encoding configurations. Based on the energy of prediction residuals and alignment scaling factors, an overall content unpredictability parameter can be estimated to compute a compression distortion factor for the video. The compression distortion factor, combined with slicing and freezing distortion factors, can be used to estimate a video quality metric for the video.