An image processing apparatus includes an abnormal portion candidate region detection unit configured to detect a candidate region of an abnormal portion based on color information of each pixel constituting an image obtained by capturing an image of an inside of a lumen of a subject, a border neighboring pixel identifying unit configured to identify a border neighboring pixel which is a pixel existing in proximity to a border of the candidate region, a feature data calculation unit configured to calculate feature data based on a pixel value of the border neighboring pixel, and an abnormal portion region distinguishing unit configured to distinguish an abnormal portion region based on the feature data.