A semiconductor device includes a fin structure over a substrate. The fin structure includes a bottom portion and a top portion. The bottom and the top portions have different materials. The device also includes a liner layer on a sidewall of the bottom portion, a dielectric layer on side surfaces of the liner layer, an interfacial layer, and a gate structure over the dielectric layer and engages the fin structure. A top surface of the liner layer extends below a bottom surface of the top portion. The interfacial layer has a first section on and directly contacting sidewall surfaces of the bottom portion and a second section on and directly contacting top and sidewall surfaces of the top portion. The gate structure includes a high-k dielectric layer and a metal gate electrode over the high-k dielectric layer. The high-k dielectric layer directly contacts the first section of the interfacial layer.