An image processing apparatus includes: a color feature data calculation unit configured to calculate color feature data of each pixel in an intraluminal image or color feature data of each small region obtained by dividing the intraluminal image into a plurality of small regions; a residue candidate distribution determination unit configured to determine, from among the color feature data, color feature data distributed on a side comparatively strong in redness to be a mucosa distribution and determine color feature data distributed on a side comparatively weak in redness to be a residue candidate distribution; and a residue distribution determination unit configured to determine, from among distributions of the color feature data determined to be the residue candidate distribution, a residue candidate distribution distributed on a side strong in yellowness with reference to the mucosa distribution to be a residue distribution.