A method of manufacturing a semiconductor device is provided. The method includes forming a channel layer and an active layer over a substrate; forming a doped epitaxial layer over the active layer; patterning the doped epitaxial layer, the active layer, and the channel layer to form a fin structure comprising a doped epitaxial fin portion, an active fin portion below the doped epitaxial fin portion, and a channel fin portion below the active fin portion; removing the doped epitaxial fin portion; and forming a gate electrode at least partially extending along a sidewall of the fin structure to form a Schottky barrier between the gate electrode and the fin structure after removing the doped epitaxial fin portion.