A method of manufacturing a semiconductor device includes forming a fin structure comprising alternately stacked first semiconductor layers and second semiconductor layers over a substrate. A sacrificial gate structure is formed over the fin structure. Spacers are formed on either side of the sacrificial gate structure. The sacrificial gate structure is removed to form a trench between the spacers. The first semiconductor layers are removed from the trench, while leaving the second semiconductor layers suspended in the trench. A self-assembling monolayer is formed on sidewalls of the spacers in the trench. Interfacial layers are formed encircling the suspended second semiconductor layers, respectively. A high-k dielectric layer is deposited at a faster deposition rate on the interfacial layers than on the self-assembling monolayer. A metal gate structure is formed over the high-k dielectric layer.