An image sensor may include a pixel array having image pixels for capturing image data and phase detection pixels for gathering phase information during automatic focusing operations. Phase detection pixels may form phase detection pixel pairs having first and second pixels with different angular responses. The first and second pixels may have color filters of the same color or may have color filters of different colors. The phase detection pixel pairs may be isolated from other phase detection pixel pairs in the array or may be arranged consecutively in a line. The phase detection pixels may, for example, be provided with color filters to match the color filter pattern of the pixel array. Processing circuitry may adjust red and green pixel signals from a phase detection pixel pair having a red and green color filter and may subsequently determine a phase difference using the adjusted pixel signals.