A video encoder performs multiple coding passes on coefficients in a coefficient block. During each coding pass, the video encoder encodes a different set of syntax elements for coefficients in the coefficient block. A video decoder uses the syntax elements for a coefficient to determine the value of the coefficient. When the video encoder performs a coding pass, the video encoder selects a coding context for a syntax element based at least in part on a syntax element generated in an earlier coding pass for a non-causal coefficient and based at least in part on a value of a syntax element generated during the current coding pass for a causal coefficient. The video encoder entropy encodes the syntax element based on the selected coding context. A video decoder performs a similar series of coding passes in which the video decoder selects coding contexts and entropy decodes the syntax elements.