A lithographic apparatus applies a device pattern at multiple fields across a substrate. A height map is decomposed into a plurality of components. A first height map component represents topographical variations associated with the device pattern. One or more further height map components represent other topographical variations. Using each height map component, control set-points are calculated according to a control algorithm specific to each component. The control set-points calculated for the different height map components are then combined and used to control imaging of the device pattern to the substrate. The specific control algorithms can be different from one another, and may have differing degrees of nonlinearity. The combining of the different set-points can be linear. Focus control in the presence of device-specific topography and other local variations can be improved.