A device includes a substrate, insulation regions extending into the substrate, and a semiconductor fin higher than top surfaces of the insulation regions. The semiconductor fin has a first lattice constant. A semiconductor region includes sidewall portions on opposite sides of the semiconductor fin, and a top portion over the semiconductor fin. The semiconductor region has a second lattice constant different from the first lattice constant. A strain buffer layer is between and contacting the semiconductor fin and the semiconductor region. The strain buffer layer includes an oxide.