An image processing apparatus includes an illumination unit in which illumination axes of first and second illumination light substantially coincide with an imaging axis and which illuminates an object with the first and second illumination light alternately, the first and second illumination light being respectively polarized in a first direction and a second direction that crosses the first direction; a splitter that divides returning light into at least two returning light components; and first and second polarization imaging devices that receive the returning light components. The first polarization imaging device obtains first and second polarization images polarized in the first direction while the object is being illuminated with the first and second illumination light, respectively. The second polarization imaging device obtains third and fourth polarization images polarized in the second direction while the object is being illuminated with the first and second illumination light, respectively.