A method of manufacturing a semiconductor device includes providing a substrate structure including a semiconductor substrate, an interlayer dielectric layer on the semiconductor substrate, multiple trenches extending through the interlayer dielectric layer to the semiconductor substrate and having a first trench of a PMOS device and a second trench of an NMOS device, and a high-k dielectric layer on sidewalls and a bottom of the trenches. The method also includes forming a semiconductor layer filling the trenches, removing the semiconductor layer in the first trench, forming a PMOS work function adjustment layer in the first trench and a metal electrode layer on the PMOS work function adjustment layer in the first trench, removing the semiconductor layer in the second trench, and forming an NMOS work function adjustment layer in the second trench and a metal electrode layer on the NMOS work function adjustment layer in the second trench.