A video compression algorithm for converting a digital video stream having video content into a compressed video bitstream, includes rate control to control the size of the compressed video bitstream. The rate control takes into account two or more of (1) detected scene changes, (2) detected repeated fields resulting from inverse telecine processing, (3) characterized video content complexity, and (4) a determined level of a model of a bit buffer associated with a decoder adapted to decode the compressed video bitstream. In preferred embodiments, the rate control takes into account two or more of these four factors in an integrated manner to change the allocation of bits for encoding different portions of the video stream.