A video processing device includes a rate control module to determine more accurate initial quantization parameters at each scene switching point and to adjust the QP parameters in response to scene changes using a sum of variances metric and an estimated picture encoding cost metric from a coding complex estimation block. To determine a first quantization parameter set, a sum of variances metric and an estimated picture encoding cost metric for an initial set pictures of a video stream are used. A bit allocation module is to set a target bit allocation for infra-encoded pictures as substantially proportional to the sum of variances metric and substantially inversely proportional to the estimated picture encoding cost metric, and set a target bit allocation for forward predictive and bi-predictive pictures as substantially proportional to the estimated picture encoding cost metric and substantially inversely proportional to the sum of variances metric.