A video decoder for decoding multi-layer video data can be configured to maintain a decoded picture buffer (DPB) for storing reference pictures for a plurality of layers, wherein the plurality of layers comprise at least a first layer and a second layer; prior to decoding a current picture of an access unit of the first layer, perform a picture output and removal process on the DPB, wherein to perform the picture output and removal process on the DPB, the one or more processors are further configured to remove from the DPB only decoded pictures that belong to the first layer; and after removing a last decoding unit of the current picture from a coded picture buffer (CPB), perform a picture bumping process across all layers of the DPB.