In a method of manufacturing a semiconductor device, first and second gate structures are formed. The first (second) gate structure includes a first (second) gate electrode layer and first (second) sidewall spacers disposed on both side faces of the first (second) gate electrode layer. The first and second gate electrode layers are recessed and the first and second sidewall spacers are recessed, thereby forming a first space and a second space over the recessed first and second gate electrode layers and first and second sidewall spacers, respectively. First and second protective layers are formed in the first and second spaces, respectively. First and second etch-stop layers are formed on the first and second protective layers, respectively. A first depth of the first space above the first side wall spacers is different from a second depth of the first space above the first gate electrode layer.