An integrated circuit structure includes a substrate; a channel layer over the substrate, wherein the channel layer is formed of a first III-V compound semiconductor material; a highly doped semiconductor layer over the channel layer; a gate dielectric penetrating through and contacting a sidewall of the highly doped semiconductor layer; and a gate electrode on a bottom portion of the gate dielectric. The gate dielectric includes a sidewall portion on a sidewall of the gate electrode.