A semiconductor device includes active regions on a semiconductor substrate, gate structures on separate, respective active regions, and source/drain regions in the semiconductor substrate on opposite sides of separate, respective gate structures. Each separate gate structure includes a sequential stack of a high dielectric layer, a first work function metal layer, a second work function metal layer having a lower work function than the first work function metal layer, and a gate metal layer. First work function metal layers of the gate structures have different thicknesses, such that the gate structures include a largest gate structure where the first work function metal layer of the largest gate structure has a largest thickness of the first work function metal layers. The largest gate structure includes a capping layer on the high dielectric layer of the largest gate structure, where the capping layer includes one or more impurity elements.