A semiconductor device includes a Fin FET device. The Fin FET device includes a first fin structure extending in a first direction and protruding from an isolation insulating layer, a first gate stack including a first gate electrode layer and a first gate dielectric layer, covering a portion of the first fin structure and extending in a second direction perpendicular to the first direction, and a first source and a first drain, each including a first stressor layer disposed over the first fin structure. The first fin structure and the isolation insulating layer are disposed over a substrate. A height Ha of an interface between the first fin structure and the first stressor layer measured from the substrate is greater than a height Hb of a lowest height of the isolation insulating layer measured from the substrate.