Defects are detected using surface topography data. The defects may be detected by determining topography characteristics within a region of interest on a sample, and the same topography characteristics of at least one reference surface. By comparing the topography characteristics in the region of interest for the sample and reference surface, common pattern structures may be removed, leaving only variations, which may be used to identify the presence of defects. For example, thresholds may be used to identify variations in the topography characteristics as defect candidates. Defects may be identified based on, e.g., size, height, shape, texture, etc. of candidate defects. In some implementations, rather than using a reference surface, the topography characteristic of the surface within the region of interest may be inspected based on prior knowledge of a required surface topography for the region of interest to determine if a defect is present.