In a method of manufacturing a semiconductor device, a fin structure protruding from a first isolation insulating layer is formed. A second isolation insulating layer made of different material than the first isolation insulating layer is formed so that a first upper portion of the fin structure is exposed. A dummy gate structure is formed over the exposed first upper portion of the first fin structure. The second isolation insulating layer is etched by using the dummy gate structure as an etching mask. The dummy gate structure is removed so that a gate space is formed. The second isolation insulating layer is etched in the gate space so that a second upper portion of the fin structure is exposed from the first isolation insulating layer. A gate dielectric layer and a gate electrode layer are formed over the exposed second portion of the fin structure.