Embodiments of mechanisms for forming a semiconductor device are provided. The semiconductor device includes a semiconductor substrate and a first gate stack. The first gate stack includes a gate dielectric layer, a first work function metal layer and a second work function metal layer directly on the first work function metal layer. The second work function metal layer and the first work function metal layer have the same metal element. The semiconductor device also includes a second gate stack. The second gate stack includes a gate dielectric layer, a barrier layer and a second work function metal layer. The second work function metal layer and the barrier layer do not have the same metal element. A first thickness of the second work function metal layer of the first gate stack is larger than a second thickness of the second work function metal layer of the second gate stack.