A method includes forming Shallow Trench Isolation (STI) regions in a semiconductor substrate and a semiconductor strip between the STI regions. The method also include replacing a top portion of the semiconductor strip with a first semiconductor layer and a second semiconductor layer over the first semiconductor layer. The first semiconductor layer has a first germanium percentage higher than a second germanium percentage of the second semiconductor layer. The method also includes recessing the STI regions to form semiconductor fins, forming a gate stack over a middle portion of the semiconductor fin, and forming gate spacers on sidewalls of the gate stack. The method further includes forming fin spacers on sidewalls of an end portion of the semiconductor fin, recessing the end portion of the semiconductor fin, and growing an epitaxial region over the end portion of the semiconductor fin.