Disclosed is method of decoding, from a video bitstream, a transform unit containing at least one chroma residual coefficient array associated with a single chroma channel. The method determines a size of the transform unit related to a hierarchical level of the transform unit in a corresponding coding unit, and identifies a maximum number of inverse transforms according to the determined size. The method decodes from the video bitstream the at least one chroma residual coefficient array using the identified maximum number of transforms, selects an inverse transform for the decoded chroma residual coefficient arrays, the inverse transform being selected from a predetermined set of inverse transforms, and applies the selected inverse transform to each of the chroma residual coefficient arrays to decode chroma residual samples for the chroma channel of the transform unit. A similar encoding method is also disclosed.