A method of manufacturing a semiconductor device includes: receiving a semiconductor structure, the semiconductor structure including: a fin structure; a dummy gate across over the fin structure to define a channel region of the fin structure; and a dummy dielectric layer separating the channel region of the fin structure from the dummy gate; removing the dummy gate and the dummy dielectric layer to expose the channel region of the fin structure; and forming a doped interfacial layer covering the channel region of the fin structure, in which the doped interfacial layer includes a dopant selected from the group consisting of Al, Hf, La, Sc, Y and a combination thereof.