An NFET containing a first high-k dielectric portion and a PFET containing a second high-k gate dielectric portion are formed on a semiconductor substrate. A gate sidewall nitride is formed on the gate of the NFET, while the sidewalls of the PFET remain free of the gate sidewall nitride. An oxide spacer is formed directly on the sidewalls of a PFET gate stack and on the gate sidewall nitride on the NFET. After high temperature processing, the first and second dielectric portions contain a non-stoichiometric oxygen deficient high-k dielectric material. The semiconductor structure is subjected to an anneal in an oxygen environment, during which oxygen diffuses through the oxide spacer into the second high-k dielectric portion. The PFET comprises a more stoichiometric high-k dielectric material and the NFET comprises a less stoichiometric high-k dielectric material. Threshold voltages of the PFET and the NFET are optimized by the present invention.