A video coding device, such as a video encoder or a video decoder, may be configured to code a duration between coded picture buffer (CPB) removal time of a first decoding unit (DU) in an access unit (AU) and a second DU, wherein the second DU is subsequent to the first DU in decoding order and in the same AU as the first DU. The video coding device may further determine a removal time of the DU based at least on the coded duration. The coding device may also code a sub-picture timing supplemental enhancement information (SEI) message associated with the first DU. The video coding device may further determine a removal time of the DU based at least in part on the sub-picture timing SEI message.