An image processing device for an endoscope outputs a video signal to an image display part that displays an observation image based on the video signal. The image display part is configured such that a first side of a display screen that displays the observation image is shorter than a second side intersecting with the first side, and is configured to be set in both a first setting state in which the first side is along a vertical direction and a second setting state in which the second side is along the vertical direction. The image processing device includes: a setting state recognition part configured to recognize the setting state; and a video signal generation part configured to generate the video signal such that the subject image in the observation image has an orientation corresponding to the setting state of the image display part.