A device includes insulation regions over portions of a semiconductor substrate, and a III-V compound semiconductor region over top surfaces of the insulation regions, wherein the III-V compound semiconductor region overlaps a region between opposite sidewalls of the insulation regions. The III-V compound semiconductor region includes a first and a second III-V compound semiconductor layer formed of a first III-V compound semiconductor material having a first band gap, and a third III-V compound semiconductor layer formed of a second III-V compound semiconductor material between the first and the second III-V compound semiconductor layers. The second III-V compound semiconductor material has a second band gap lower than the first band gap. A gate dielectric is formed on a sidewall and a top surface of the III-V compound semiconductor region. A gate electrode is formed over the gate dielectric.