A method includes forming a gate stack over top surfaces of a semiconductor strip and insulation regions on opposite sides of the semiconductor strip. The insulation regions include first portions overlapped by the gate stack, and second portions misaligned from the gate stack. An end portion of the semiconductor strip is etched to form a recess, wherein the recess is located between the second portions of the insulation regions. An epitaxy is performed to grow a source/drain region from the recess. After the epitaxy, a recessing is performed to recess the second portions of the insulation regions, with the second portions of the insulation regions having first top surfaces after the first recessing. After the recessing, a dielectric mask layer is formed on the first top surfaces of the second portions of the insulation regions, wherein the dielectric mask layer further extends on a sidewall of the gate stack.