A high mobility semiconductor layer is formed over a semiconductor substrate. An interfacial oxide layer is formed over the high mobility semiconductor layer. A high dielectric constant (high-k) dielectric layer is formed over the interfacial oxide layer. A stack is formed over the high-k dielectric layer. The stack comprises a lower metal layer, a scavenging metal layer comprising a scavenging metal, and an upper metal layer formed on the scavenging metal layer. A Gibbs free energy change of a chemical reaction, in which an atom constituting the high mobility semiconductor layer that directly contacts the interfacial oxide layer combines with a metal oxide material comprising the scavenging metal and oxygen to form the scavenging metal in elemental form and oxide of the atom constituting the high mobility semiconductor layer that directly contacts the interfacial oxide layer, is positive. A gate electrode and a gate dielectric are formed.