A semiconductor device has a gate with three conductive layers over a high K gate dielectric. The first layer is substantially oxygen free. The work function is modulated to the desired work function by a second conductive layer in response to subsequent thermal processing. The second layer is a conductive oxygen-bearing metal. With sufficient thickness of the first layer, there is minimal penetration of oxygen from the second layer through the first layer to adversely impact the gate dielectric but sufficient penetration of oxygen to change the work function to a more desirable level. A third layer, which is metallic, is deposited over the second layer. A polysilicon layer is deposited over the third layer. The third layer prevents the polysilicon layer and the oxygen-bearing layer from reacting together.