Mask data is analyzed for the presence of a notch. A notch candidate on a polygon boundary of mask data is defined as a plurality of line segments that includes an initial line segment, a final line segment and at least two line segments therebetween. The initial and final line segments define adjacent edges of the notch candidate. A direction of each line segment is a direction of travel from the initial line segment to the final line segment.