An integrated circuit structure includes a first semiconductor strip, first isolation regions on opposite sides of the first semiconductor strip, and a first epitaxy strip overlapping the first semiconductor strip. A top portion of the first epitaxy strip is over a first top surface of the first isolation regions. The structure further includes a second semiconductor strip, wherein the first and the second semiconductor strips are formed of the same semiconductor material. Second isolation regions are on opposite sides of the second semiconductor strip. A second epitaxy strip overlaps the second semiconductor strip. A top portion of the second epitaxy strip is over a second top surface of the second isolation regions. The first epitaxy strip and the second epitaxy strip are formed of different semiconductor materials. A bottom surface of the first epitaxy strip is lower than a bottom surface of the second epitaxy strip.