A model calculating device calculates a mammary gland normal architecture model that scatters in a fan shape from the nipple toward the greater pectoral muscle with respect to the X-ray image of the breast. An orientation extracting device extracts linear components orientations of a region image texture that form a shape of a shadow in the breast X-ray image using a Gabor filter. A lesion determining device compares the mammary gland orientation in the normal architecture model calculated by the model calculating device and the orientation extracted by the orientation extracting device with respect to a region of interest in the X-ray image of the breast including architectural distortion candidates of the mammary gland detected by a preprocessing device to calculate a feature quantity based on a difference between the orientations and determines whether the candidates are an architectural distortion of the mammary gland based on the feature quantity.