A method determines a plurality of interpolation filters for use in interpolating sub-pel values for a temporal prediction process of video content. The plurality of interpolation filters are designed based on sub-pel offsets for two reference blocks. The method determines a first sub-pel offset for a first reference block for the unit of video content and determines a second sub-pel offset for a second reference block for a unit of video content. A set of interpolation filters is determined to interpolate a set of sub-pel pixel values for use in the temporal prediction process for the unit of video content. The set of interpolation filters is designed for the first sub-pel offset and the second sub-pel offset and is used to interpolate a first sub-pel pixel value for the first reference block and a second sub-pel value for the second reference block.