A method and apparatus for determining missing color pixel values in image data. Pixel values are interpolated in a horizontal or a vertical direction based on gradient scores and optionally chroma and local statistical scores. Several techniques for refining the interpolation direction determination are also disclosed, including applying a low-pass filter, applying one or more digital filters, and growing an interpolation direction region to nearby pixels. A technique for interpolating corner pixel values is also disclosed.