A semiconductor gate stack comprising a silicon oxide based gate dielectric and a doped semiconductor material is formed on a semiconductor substrate. A high-k material metal gate electrode comprising a high-k gate dielectric and a metal gate portion is also formed on the semiconductor substrate. Oxygen-impermeable dielectric spacers are formed on the sidewalls of the semiconductor gate stack and the high-k material metal gate stack. The oxygen-impermeable dielectric spacer on the semiconductor gate stack is removed, while the oxygen impermeable dielectric spacer on the high-k material metal gate electrode is preserved. A low-k dielectric spacer is formed on the semiconductor gate stack, which provides a low parasitic capacitance for the device employing the semiconductor gate stack.