A semiconductor device and a method of fabricating the same are described. A substrate having a PMOS area and an NMOS area is provided. A high-k layer is formed on the substrate. A first cap layer is formed on the high-k layer in the PMOS area, and a second cap layer is formed on the high-k layer in the NMOS area, wherein the first cap layer is different from the second cap layer. A metal layer and a polysilicon layer are sequentially formed on the first and second cap layers. The polysilicon layer, the metal layer, the first cap layer, the second cap layer and the high-k layer are patterned to form first and second gate structures respectively in the PMOS and NMOS areas. First source/drain regions are formed in the substrate beside the first gate structure. Second source/drain regions are formed in the substrate beside the second gate structure.