An endoscope insertion direction detecting device includes a classification section for performing, on a scene of an endoscope image of a moving image picked up by an endoscope inserted into a body cavity, classification into classes for a plurality of different feature values relating to detection of an endoscope insertion direction in the body cavity; and an insertion direction computing section, provided for each of the classes for the feature values, into which the classification is performed, for computing an insertion direction of the endoscope.