A semiconductor device includes a semiconductor substrate; a gate dielectric layer disposed on the semiconductor substrate; a gate conductive layer doped with impurities selected from nitrogen, carbon, silicon, germanium, fluorine, oxygen, helium, neon, xenon or a combination thereof on the gate dielectric layer; and source/drain doped regions formed adjacent to the gate conductive layer in the semiconductor substrate, wherein the source and drain doped regions are substantially free of the impurities doped into the gate conductive layer. These impurities reduce the diffusion rates of the N-type of P-type dopants in the gate conductive layer, thereby improving the device performance.