A method includes forming a first protruding semiconductor fin and a dummy fin protruding higher than top surfaces of isolation regions. The first protruding semiconductor fin is parallel to the dummy fin, forming a gate stack on a first portion of the first protruding semiconductor fin and a second portion of the dummy fin. The method further includes recessing a third portion of the first protruding semiconductor fin to form a recess, recessing an fourth portion of the dummy fin to reduce a height of the fourth portion of the dummy fin, and forming an epitaxy semiconductor region in the recess. The epitaxy semiconductor region is grown toward the dummy fin.