This disclosure describes techniques for performing entropy encoding and decoding of video coefficients using a joint context model shared between transform units having different sizes. For example, the joint context model may be shared between transform units having a first size of 32×32 and transform units having a second size of 16×16. Performing entropy coding using a joint context model shared between transform units having different sizes may reduce an amount of memory necessary to store contexts and probabilities, and reduce computational costs of maintaining context models. In one example, the joint context model may be shared between transform units having the first size with coefficients zeroed out to generate a retained coefficient block having the second size and transform units having the second size. In another example, the joint context model may be shared between transform units having the first size and transform units having the second size.