An image processing device for an endoscope in which an operation of bending a distal end portion only within one endoscope bending plane including an axis of the distal end portion is allowed, the image processing device having a processor configured to acquire information of a course along which the endoscope travels in the luminal structure, the course being set based on a three-dimensional image indicating the luminal structure, and calculate a first rotation angle of the endoscope at a first branch in the course, the first rotation angle being an angle to match the one endoscope bending plane where the distal end portion is allowed to bend with a plane of the first branch, by a operation of rotation.