Down-sampling of an image may be performed in the DCT domain. Transform matrices are obtained for down-sampling a DCT image of size M×N to a down-sampled DCT image of size I×J. The transform matrices may be used to down-sample the DCT image directly in the DCT domain. A spatial domain down-sampling method is selected and applied to the DCT image to produce a down-sampled DCT reference image. The transform matrices are selected by solving an optimization problem, leading to transform matrices which achieve a desired trade-off between the visual quality of images obtained using the transform matrices and the computational complexity associated with using the transform matrices. The visual quality is a measure of the difference between the down-sampled DCT image obtained using the transform matrices and the visual quality of the DCT reference image obtained using a spatial domain down-sampling method.