A method includes forming a semiconductor fin protruding higher than top surfaces of isolation regions. A top portion of the semiconductor fin is formed of a first semiconductor material. A semiconductor cap layer is formed on a top surface and sidewalls of the semiconductor fin. The semiconductor cap layer is formed of a second semiconductor material different from the first semiconductor material. The method further includes forming a gate stack on the semiconductor cap layer, forming a gate spacer on a sidewall of the gate stack, etching a portion of the semiconductor fin on a side of the gate stack to form a first recess extending into the semiconductor fin, recessing the semiconductor cap layer to form a second recess directly underlying a portion of the gate spacer, and performing an epitaxy to grow an epitaxy region extending into both the first recess and the second recess.