A workpiece inspection and defect detection system includes a light source, a lens that inputs image light arising from a surface of a workpiece, and a camera that receives imaging light transmitted along an imaging optical path. The system utilizes images of workpieces acquired with the camera as training images to train a defect detection portion to detect defect images that include workpieces with defects. Anomaly detector classification characteristics are determined based on features of the training images. Run mode images of workpieces are acquired with the camera, and based on determined features from the images, the anomaly detector classification characteristics are utilized to determine if the images of the workpieces are classified as anomalous. In addition, the defect detection portion determines if images are defect images that include workpieces with defects and for which additional operations may be performed (e.g., metrology operations for measuring dimensions of the defects, etc.)