In a method of manufacturing a semiconductor device, a sacrificial gate structure including sacrificial gate electrode is formed over a substrate. A first dielectric layer is formed over the sacrificial gate structure. A second dielectric layer is formed over the first dielectric layer. The second and first dielectric layers are planarized and recessed, and an upper portion of the sacrificial gate structure is exposed. A third dielectric layer is formed over the exposed sacrificial gate structure and over the first dielectric layer. A fourth dielectric layer is formed over the third dielectric layer. The fourth and third dielectric layers are planarized, and the sacrificial gate electrode is exposed and part of the third dielectric layer remains on the recessed first dielectric layer. The recessing the first dielectric layer comprises a first etching operation and a second etching operation using a different etching as from the first etching operation.