A defect image classification apparatus includes a control unit that selects images obtained from at least some detectors among a plurality of detectors, associated with kinds of defects to be a classification result of an automatic defect classification processing unit, as images displayed initially on a display unit. The control unit associates the kinds of the defects and the images displayed initially on the display unit, on the basis of a switching operation log when a user classifies images of defects determined previously as the same kinds as the kinds of the defects determined by the automatic defect classification processing unit.