A semiconductor device includes a fin structure protruding from a first isolation insulating layer provided over a substrate, a gate dielectric layer disposed over a channel region of the fin structure, a gate electrode layer disposed over the gate dielectric layer, a base semiconductor epitaxial layer disposed over a source/drain region of the fin structure, and a cap semiconductor epitaxial layer disposed over the base semiconductor epitaxial layer. The cap semiconductor epitaxial layer has a different lattice constant than the base semiconductor epitaxial layer, and a surface roughness of the cap semiconductor epitaxial layer along a source-to-drain direction is greater than zero and smaller than a surface roughness of the base semiconductor epitaxial layer along the source-to-drain direction.