An imager may include depth sensing pixels that provide an asymmetrical angular response to incident light. The depth sensing pixels may each include a substrate region formed from a photosensitive portion and a non-photosensitive portion. The depth sensing pixels may include mechanisms that prevent regions of the substrate from receiving incident light. Depth sensing pixel pairs may be formed from depth sensing pixels that have different asymmetrical angular responses. Each of the depth sensing pixel pairs may effectively divide the corresponding imaging lens into separate portions. Depth information for each depth sensing pixel pair may be determined based on the difference between output signals of the depth sensing pixels of that depth sensing pixel pair. The imager may be formed from various combinations of depth sensing pixel pairs and color sensing pixel pairs arranged in a Bayer pattern or other desired patterns.