A system and method for processing video data having a first random access point (RAP) picture. An access unit having a first random access point (RAP) picture is received in a video bitstream. A subsequent access unit having a RAP picture that does not initialize the hypothetical reference decoder (HRD) is also received and, based on one or more random access skipped leading (RASL) pictures for the subsequent access unit not being present in the video bitstream, a picture buffer removal time for a picture buffer is shifted earlier based on a picture buffer removal delay offset.