The present disclosure relates to an image processing apparatus, an image processing method, a program, and an endoscope system capable of obtaining an image with high color discrimination. A histogram generation unit generates histograms of pixel values for an input image regarding luminance Y, saturation S, and hue H. A histogram correction unit subtracts components of the saturation, the luminance, and the hue of a portion determined to be a foreign object by a foreign object detection unit from the histograms regarding the saturation, the luminance, and the hue generated by the histogram generation unit and then weights by a value of a standard histogram. The present disclosure may be applied to the endoscope system which performs color conversion on an image input from a camera head through a scope, for example.