The present invention teaches a filter process used in intra or inter prediction of pixel blocks. A first, optionally interpolation, filter is applied in the first filter process to get filtered, optionally interpolated, pixel values. A differential filter and an adaptive gain are utilized in the second process for improving the prediction performance. The adaptivity of the gain can be made even on block basis, allowing a fine tuning of the pixel prediction and/or a fine tuning of pixel rotation and zooming. Alternatively, a combined one-step filter process using the interpolation filter, the differential filter and the adaptive gain is applied to the pixel values.