An overlay measurement apparatus has a polarized light source for illuminating a sample with a polarized light beam and an optical system to capture light that is scattered by the sample. The optical system includes a polarizer for transmitting an orthogonal polarization component that is orthogonal to a polarization direction of the polarized light beam. A detector measures intensity of the orthogonal polarization component. A processing unit is connected to the detector, and is arranged to process the orthogonal polarization component for overlay metrology measurement using asymmetry data derived from the orthogonal polarization component.