An imaging system may be provided with a backside illuminated pixel array that includes imaging pixels and rows of auto-focus pixels. The imaging pixels may generate image data and the auto-focus pixels may generate application-specific data. The imaging pixels may be provided with an array of color filters. The auto-focus pixels may be provided with rows of color filters that are offset with respect to the imaging pixel color filters. The auto-focus pixel color filters may include clear color filters and opaque color filters for blocking image light received at selected incident angles. The system may include an adjustable lens that focuses light onto the pixel array. The pixel array may be coupled to processing circuitry that determines whether an image is focused based on the application-specific data and may adjust the lens in response to determining that the image is not focused.