This disclosure describes techniques for transforming residual blocks of video data. In particular, a plurality of different transforms selectively applied to the residual blocks based on the prediction mode of the video blocks. At least a portion of the plurality of transforms are separable directional transform specifically trained for a corresponding prediction mode to provide better energy compaction for the residual blocks of the given prediction mode. Using separable directional transforms offers the benefits of lower computation complexity and storage requirement than use of non-separable directional transforms. Additionally, a scan order used to scan the coefficients of the residual block may be adjusted when applying separable directional transforms. In particular, the scan order may be adjusted based on statistics associated with one or more previously coded blocks to better ensure that non-zero coefficients are grouped near the front of the one-dimensional coefficient vector to improve the effectiveness of entropy coding.