An optical response measuring device is provided with a light source, first and second wavelength conversion elements and a light intensity sensor array. The light source generates a pair of light beams including light beams of first and second wavelengths, and the first wavelength conversion element generates measurement light. The measurement light is irradiated on an object for measurement and a detection light having first phase and second phase is obtained in response to this irradiation. A reference light that carries the phase of the pair of the pair light beams and the detection light both pass through a second wavelength conversion element to obtain a modulated reference light have first and second intensities. The first and second local intensities are then measured by the light intensity sensor array.