A device isolation region is made of a silicon oxide film embedded in a trench, an upper portion thereof is protruded from a semiconductor substrate, and a sidewall insulating film made of silicon nitride or silicon oxynitride is formed on a sidewall of a portion of the device isolation region which is protruded from the semiconductor substrate. A gate insulating film of a MISFET is made of an Hf-containing insulating film containing hafnium, oxygen and an element for threshold reduction as main components, and a gate electrode that is a metal gate electrode extends on an active region, the sidewall insulating film and the device isolation region. The element for threshold reduction is a rare earth or Mg when the MISFET is an n-channel MISFET, and the element for threshold reduction is Al, Ti or Ta when the MISFET is a p-channel MISFET.