A data processing unit acquires a review image including a pattern defect on a substrate, compares the review image with a reference image thereby to extract a defect image, the reference image including no pattern defect, and performs an alignment between the review image and a self-layer design pattern image which is generated from design data belonging to the identical layer in a region corresponding to the review image. The data processing unit, then, based on result of the alignment, generates an another-layer design pattern image which is generated from design data belonging to another layer in the region corresponding to the review image, and, based on a synthesized image of the defect image and the another-layer design pattern image, determines the relative position relationship between the pattern defect and a pattern belonging to another layer, and judges the criticality based on the relative position relationship.