Methods and devices for encoding and decoding data using adaptive transform domain filtering are described. The encoder determines a set of transform domain filter coefficients to be applied to a transform domain prediction. The filtering may, in some cases, also apply to quantized transform domain coefficients. Rate-distortion optimization may be used to determine the optimal filter coefficients on a frame-based, coding-unit-basis, or other basis.