A ranging processing device includes: a four-phase ranging operation unit that performs an operation to calculate depth indicating a distance to an object by using all eight detection signals two of which are detected for each of irradiated light of first to fourth phases; a two-phase ranging operation unit that performs the operation to calculate the depth indicating the distance to the object by alternately using four detection signals based on the irradiated light of the first phase and the irradiated light of the second phase and four detection signals based on the irradiated light of the third phase and the irradiated light of the fourth phase among the eight detection signals; and a condition determination unit that makes condition determination based on the detection signals and switch between the four-phase ranging operation unit and the two-phase ranging operation unit to be used.