A semiconductor device and method of forming the same includes a substrate having a NMOS region and a PMOS region. The method includes forming a dummy gate structure having a stacked sacrificial dielectric layer and a sacrificial gate material layer on the NMOS and PMOS regions. The method further includes concurrently removing the stacked sacrificial dielectric layer and a sacrificial gate material layer to form a groove, and forming a high-K dielectric layer and a first metal gate layer in the grove. The method also includes forming a hard mask over the NMOS region, removing the first metal gate layer and the high-K dielectric layer in the PMOS region to form a channel groove, forming a second high-K dielectric layer and a second metal gate layer in the channel grove, and removing the hard mask. The work function metal layer in the NMOS and PMOS regions can be independently controlled.