A semiconductor device may include a semiconductor substrate, a diffusion layer provided over the semiconductor substrate, source and drain diffusion regions provided in upper regions of the diffusion layer, a gate insulating film provided over the source and drain diffusion regions and the diffusion layer, a gate electrode provided on the gate insulating film and positioned over the diffusion layer, a passivation film provided over the gate insulating film and the gate electrode, an insulating film that covers the passivation film, and contact plugs that penetrate the insulating film, the passivation film, and the gate insulating film, so that the contact plugs reach the source and drain diffusion regions. The contact plugs are positioned near side walls of the gate electrode. Fluorine is implanted to the passivation film. Fluorine is diffused to a silicon-insulator interface between the gate insulating film and the diffusion layer under the gate electrode.