This disclosure describes techniques for scanning coefficients of video blocks, e.g., quantized and transformed coefficients. Rather than use conventional zig-zag scanning, the techniques of this disclosure adapt the scanning order based on statistics associated with previously coded blocks that were coded in the same prediction mode. For each prediction mode, statistics of the coefficients are stored, e.g., indicating probabilities that given coefficients are zero or non-zero. Periodically, adjustments to the scanning order can be made in order to better ensure that non-zero coefficients are grouped together and zero value coefficients are grouped together, which can improve the effectiveness of entropy coding. The techniques of this disclosure provide thresholds and threshold adjustments that can reduce the frequency that the scanning order adjustments occur, yet still achieve desired improvements in compression due to such scanning order adjustments.