In a MOS-type semiconductor device in which, on a Si substrate, a SiGe layer having a valence band edge energy value smaller than a valence band edge energy value of the first semiconductor layer and a mobility larger than a mobility of the first semiconductor layer, a Si cap layer, and an insulating layer are sequentially laminated, the problem of the shift of the absolute value of the threshold voltage toward a smaller value caused by negative fixed charges formed in or near the interface between the Si cap layer and the insulting film by diffusion of Ge is overcome by neutralizing the negative fixed charges by positive charges induced in and near the interface between the Si cap layer and the insulating film along with addition of nitrogen atoms to the semiconductor device surface by NO gas annealing and thereby shifting the threshold voltage toward a larger value.