In a method of manufacturing a semiconductor device, a fin structure having a lower fin structure and an upper fin structure disposed over the lower fin structure is formed. The upper fin structure includes first semiconductor layers and second semiconductor layers alternately stacked. The first semiconductor layers are partially etched to reduce widths of the first semiconductor layers. An oxide layer is formed over the upper fin structure. A sacrificial gate structure is formed over the upper fin structure with the oxide layer. A source/drain epitaxial layer is formed over a source/drain region of the fin structure. The sacrificial gate structure is removed to form a gate space. The oxide layer is removed to expose the second semiconductor layers in the gate space. A gate structure is formed around the second semiconductor layers in the gate space.