The disclosure relates to a method of manufacturing a polarization-modulating optical element, wherein the element causes, for light passing through the element and due to stress-induced birefringence, a distribution of retardation between orthogonal states of polarization, the method comprising joining a first component and a second component, wherein a non-plane surface of the first component being provided with a defined height profile is joined with a plane surface of the second component, whereby a mechanical stress causing the stress-induced birefringence is produced in the such formed polarization-modulating optical element.