In a method of manufacturing a semiconductor device, a gate dielectric layer is formed over a channel region made of a semiconductor material, a first barrier layer is formed on the gate dielectric layer, a second barrier layer is formed on the first barrier layer, a first work function adjustment layer is formed on the second barrier layer, the first work function adjustment layer and the second barrier layer are removed. After the first work function adjustment layer and the second barrier layer are removed, a second work function adjustment layer is formed over the gate dielectric layer, and a metal gate electrode layer is formed over the second work function adjustment layer.