One embodiment provides a method for determining mask layouts. During operation, the system can receive a design intent. Next, the system can determine a set of critical edges in the design layout, and select a first edge and a second edge. The system can then determine a first trench and a second trench using the first edge and the second edge, respectively. Note that an edge of the first trench may substantially overlap with the first edge, and an edge of the second trench may substantially overlap with the second edge. Next, the system may assign the first trench and the second trench to the first mask layout and the second mask layout, respectively. The system can then increase the first trench and the second trench, thereby improving pattern fidelity. The resulting mask layouts may be used in a double patterning process.