Methods and apparatus for decoding a compressed Higher Order Ambisonics (HOA) representation of a sound or soundfield. The method may include receiving a bit stream containing the compressed HOA representation and decoding, based on a determination that there are multiple layers, the compressed HOA representation from the bitstream to obtain a sequence of decoded HOA representations. A first subset of the sequence of decoded HOA representations is determined based only on corresponding ambient HOA components. A second subset of the sequence of decoded HOA representations is determined based on corresponding ambient HOA components and corresponding predominant sound components. For a frame k, the sequence of decoded HOA representations are represented at least in part by c ^ ~ n ⁡ ( k - 1 ) = { c ^ AMB , n ⁡ ( k - 1 ) for ⁢ ⁢ n ⁢ ⁢ in ⁢ ⁢ the ⁢ ⁢ first ⁢ ⁢ subset c ^ n ⁡ ( k - 1 ) = c ^ PS , n ⁡ ( k - 1 ) + c ^ AMB , n ⁡ ( k - 1 ) , for ⁢ ⁢ n ⁢ ⁢ in ⁢ ⁢ the ⁢ ⁢ second ⁢ ⁢ subset where ĉAMB,n(k−1) corresponds to the corresponding ambient HOA components and ĉPS,n(k−1) corresponds to the corresponding predominant sound components.