Photosensors may be formed on a front side of a semiconductor substrate. An optical refraction layer having a first refractive index may be formed on a backside of the semiconductor substrate. A grid structure including openings is formed over the optical refraction layer. A masking material layer is formed over the grid structure and the optical refraction layer. The masking material layer may be anisotropically etched using an anisotropic etch process that collaterally etches a material of the optical refraction layer and forms non-planar distal surface portions including random protrusions on physically exposed portions of the optical refraction layer. An optically transparent layer having a second refractive index that is different from the first refractive index may be formed on the non-planar distal surface portions of the optical refraction layer. A refractive interface refracts incident light in random directions, and improves quantum efficiency of the photo sensors.