A method includes forming a semiconductor fin over top surfaces of insulation regions, and forming a gate stack on a top surface and sidewalls of a middle portion of the semiconductor fin. The insulation regions include first portions overlapped by the gate stack, and second portions misaligned from the gate stack. An end portion of the semiconductor fin is etched to form a recess located between the second portions of the insulation regions. An epitaxy is performed to grow a source/drain region from the recess. After the epitaxy, a recessing is performed to recess the second portions of the insulation regions, with the second portions of the insulation regions having first top surfaces after the first recessing. A dielectric mask layer is formed on the first top surfaces of the second portions of the insulation regions. The dielectric mask layer further extends on a sidewall of the gate stack.