A method to determine a curvilinear pattern of a patterning device that includes obtaining (i) an initial image of the patterning device corresponding to a target pattern to be printed on a substrate subjected to a patterning process, and (ii) a process model configured to predict a pattern on the substrate from the initial image, generating, by a hardware computer system, an enhanced image from the initial image, generating, by the hardware computer system, a level set image using the enhanced image, and iteratively determining, by the hardware computer system, a curvilinear pattern for the patterning device based on the level set image, the process model, and a cost function, where the cost function (e.g., EPE) determines a difference between a predicted pattern and the target pattern, where the difference is iteratively reduced.