A system and method for decoding video encoded using scalable video coding. In one embodiment, a decoder for scalable video coding (SVC) includes an SVC access unit analyzer and decoding logic. The SVC access unit analyzer is configured to examine an SVC access unit prior to layered decoding of the access unit, to determine, based on the examination, what operations the access unit specifies for each layer of the decoding, and to determine, based on the determined operations to be performed for each layer of the decoding, what data to store for use by a subsequent layer of the decoding. The decoding logic is configured to decode the access unit via a plurality of decoding layers; and to store at each decoding layer, for use by a subsequent decoding layer, the data determined by the SVC access unit analyzer to be used by the subsequent decoding layer.