After forming a pure silicon oxide film on respective surfaces of an n-type well and a p-type well, an oxygen deficiency adjustment layer made of an oxide of 2A group elements, an oxide of 3A group elements, an oxide of 3B group elements, an oxide of 4A group elements, an oxide of 5A group elements or the like, a high dielectric constant film, and a conductive film having a reduction catalyst effect to hydrogen are sequentially deposited on the silicon oxide film, and the substrate is heat treated in the atmosphere containing H2, thereby forming a dipole between the oxygen deficiency adjustment layer and the silicon oxide film. Then, the conductive film, the high dielectric constant film, the oxygen deficiency adjustment layer, the silicon oxide film and the like are patterned, thereby forming a gate electrode and a gate insulating film.