Systems and methods of performing rate control in scalable video encoders for use in videoconferencing, announcements, and live video streaming to multiple participant devices having diverse bandwidths, resolutions, and/or other device characteristics. The systems and methods can accommodate different target bit rates of the multiple participant devices by operating on scalable video bitstreams in a multi-layer video format, including a base layer having one or more reference video frames, and an enhancement layer having one or more disposable non-reference, predictive video frames. By adjusting the number of disposable non-reference, predictive video frames in the enhancement layer, as well as quantization parameters for the respective base and enhancement layers, the systems and methods can accommodate the different target bit rates for the respective participant devices, while enhancing the spatial and/or temporal qualities of the base and enhancement layers in the respective video bitstreams.