A method of forming a gate structure for a semiconductor device that includes forming a non-stoichiometric high-k gate dielectric layer on a semiconductor substrate, wherein an oxide containing interfacial layer can be present between the non-stoichiometric high-k gate dielectric layer and the semiconductor substrate. At least one gate conductor layer may be formed on the non-stoichiometric high-k gate dielectric layer. The at least one gate conductor layer comprises a boron semiconductor alloy layer. An anneal process is applied, wherein during the anneal process the non-stoichiometric high-k gate dielectric layer removes oxide material from the oxide containing interfacial layer. The oxide containing interfacial layer is thinned by removing the oxide material during the anneal process.