In one embodiment, a method determines buffer parameter settings for a plurality of layers in a scalable video stream for a video. Each layer includes a respective buffer parameter setting. Then, the method applies respective buffer parameter settings to individual buffers for respective layers in the plurality of layers. The respective layers are buffered in the individual buffers according to the respective buffer parameter settings where the respective layers are combined after the buffers to form a combined bitstream including the respective layers for decoding.