An imaging system may include an image sensor with phase detection pixel groups for depth sensing or automatic focusing operations. Each phase detection pixel group may have two or more photosensitive regions covered by a single microlens so that each photosensitive region has an asymmetric angular response. The image sensor may be sensitive to both near-infrared (NIR) and visible light. Each phase detection pixel group may be designed to include light-scattering structures that increase NIR sensitivity while minimizing disruptions of phase detection and visible light performance. Deep trench isolation may be formed between adjacent photosensitive areas within the phase detection pixel group. The light-scattering structures may have a non-uniform distribution to minimize disruptions of phase detection performance.