A system includes an interference microscope having one or more optical elements arranged to image a test object to an image plane by combining test light from the test object with reference light from a reference object to form an interference pattern at the image plane, wherein the test and reference light are derived from a common broadband light source. The system includes a scanning stage configured to scan an optical path difference (OPD) between the test and reference light, a multi-element detector positioned at the image plane and configured to record the interference pattern for each of a series of OPD increments and to generate multiple interferometry signals each having a fringe carrier frequency indicative of changes in the OPD as the OPD is scanned, where there is phase diversity among the interferometry signals, and an electronic processor coupled to the multi-element detector and scanning stage and configured to process the interference signals based on the phase diversity to determine information about the OPD increments having sensitivity to perturbations to the OPD increments at frequencies greater than the fringe carrier frequency.