One embodiment relates to a method of classifying a defect on a substrate surface. The method includes scanning a primary electron beam over a target region of the substrate surface causing secondary electrons to be emitted therefrom, wherein the target region includes the defect. The secondary electrons are detected from the target region using a plurality of at least two off-axis sensors so as to generate a plurality of image frames of the target region, each image frame of the target region including data from a different off-axis sensor. The plurality of image data frames are processed to generate a surface height map of the target region, and surface height attributes are determined for the defect. The surface height attributes for the defect are input into a defect classifier. Other embodiments, aspects and features are also disclosed.