A recess detection device includes a reference value decision processor that extracts, from distance image information, pieces of reference information respectively corresponding to reference regions at different positions within a detection range with a distance from a three-dimensional sensor being within a predetermined range, and determines a reference value based on the reference information, and a presence identification processor that extracts, from the distance image information, pieces of comparison information respectively corresponding to comparison regions within the same detection range, respectively, derives comparison values based on the respective pieces of comparison information, and indicates a presence of a recess when a difference between the reference value and the comparison value exceeds a predetermined threshold.