A method of manufacturing a semiconductor device includes forming an epitaxial layer within a source/drain region of a semiconductor substrate, forming a fluorine-containing layer on the surface of the epitaxial layer, forming a metal gate structure within the gate region after the step of forming the fluorine-containing layer, forming an interlayer dielectric to cover the semiconductor substrate, the epitaxial layer and the metal gate structure, forming a contact hole penetrating the interlayer dielectric to expose a portion of the epitaxial layer, forming a metal silicide layer on or in the epitaxial layer on a bottom of the contact hole so that the fluorine-containing layer is disposed on the periphery of the metal silicide layer.