A semiconductor device includes a semiconductor substrate, an interfacial layer formed on the semiconductor substrate, a high-k dielectric layer formed on the interfacial layer, and a conductive gate electrode layer formed on the high-k dielectric layer. At least one of the high-k dielectric layer and the interfacial layer is doped with: a first dopant species, a second dopant species, and a third dopant species. The first dopant species and the second dopant species form a plurality of first dipole elements having a first polarity. The third dopant species forms a plurality of second dipole elements having a second polarity, and the first and second polarities are opposite.