In an example, a method of coding video data includes coding one or more non-video coding layer (VCL) network abstraction layer (NAL) units of a layer of a multi-layer bitstream of video data, where the one or more non-VCL NAL units contain an SEI message having an SEI payload type. The method also includes determining one or more syntax values of the multi-layer bitstream to which the SEI message applies based on the SEI payload type.