A method and system for fracturing or mask data preparation is disclosed in which the central core portion of a diagonal pattern is fractured using overlapping variable shaped beam (VSB) shots, and an outer portion of the diagonal pattern is fractured using non-overlapping VSB shots. A transition region is interposed between the central core and outer pattern portions, and transition region shots are generated so as to produce in the transferred pattern a smooth transition in pattern characteristics such as line edge roughness or period of waviness, from the central core portion of the pattern to the outer portion of the pattern. A pattern determined by the transition region shots is then compared to a reticle pattern created using conventional non-overlapping VSB shots. Methods for forming a semiconductor device layout pattern on a reticle or substrate are also disclosed.