Methods, devices and systems for configuring different access units in scalable video coding are described. In one example aspect, a method of video processing include performing a conversion between a video including one or more pictures in one or more video layers and a bitstream of a video. The bitstream includes a coded video sequence that includes one or more access units. The bitstream conforms to a format rule that specifies that each access unit of the one or more access units that is a gradual decoding refresh access unit includes exactly one picture for each video layer present in the coded video sequence.