According to one embodiment, a design layout highly likely to be a dangerous point in a lithography process is set, a coherence map kernel for generating the mask layout is set with respect to the set design layout, the coherence map is created based on the set coherence map kernel and the set design layout, the auxiliary pattern is extracted from the created coherence map and shaped to generate the mask layout, a cost function COST for evaluating an optimization degree of the mask layout is defined, the generated mask layout is evaluated using the cost function, and at least one of parameters of the coherence map kernel and parameters in extracting and shaping the auxiliary pattern from the coherence map are changed until the mask layout evaluated using the cost function is optimized.