An encoder receives a first image of a first spatial resolution and a second image of a second spatial resolution, wherein both the first image and the second image represent the same scene and the second spatial resolution is higher than the first spatial resolution. A filter is selected to up-sample the first image to a third image with a spatial resolution same as the second spatial resolution. The filtering coefficients for the up-sampling filter are computed by minimizing an error measurement (e.g., MSE) between pixel values of the second image and the third image. The computed set of filtering coefficients is signaled to a receiver (e.g., as metadata). A decoder receives the first image (or its approximation) and the metadata, and may up-sample the first image using the same filter and optimally selected filtering coefficients as those derived by the encoder.