Surface image data of a non-defective sample substrate is acquired, and surface image data of a substrate to be inspected is acquired. Differences between gradation values are calculated for pixels of the surface image data of the substrate to be inspected and corresponding pixels of the surface image data of the sample substrate. A constant value is added to the difference between gradation values of each pixel. In the case where the value acquired by addition is in a predetermined allowable range, it is determined that there is no defect for the substrate to be inspected. In the case where the value acquired by addition is outside of the allowable range, it is determined that the substrate to be inspected is defective. A defect in appearance on the substrate to be inspected is detected based on a pixel of which the value is outside of the allowable range.