A method of manufacturing a semiconductor device, a plurality of fin structures are formed over a semiconductor substrate. The fin structures extend along a first direction and are arranged in a second direction crossing the first direction. A plurality of sacrificial gate structures extending in the second direction are formed over the fin structures. An interlayer dielectric layer is formed over the plurality of fin structures between adjacent sacrificial gate structures. The sacrificial gate structures are cut into a plurality of pieces of sacrificial gate structures by forming gate end spaces along the second direction. Gate separation plugs are formed by filling the gate end spaces with two or more dielectric materials. The two or more dielectric materials includes a first layer and a second layer formed on the first layer, and a dielectric constant of the second layer is smaller than a dielectric constant of the first layer.