Embodiments of mechanisms for forming a semiconductor device are provided. The semiconductor device includes a substrate. The semiconductor device also includes a first fin and a second fin over the substrate. The semiconductor device further includes a first gate electrode and a second gate electrode traversing over the first fin and the second fin, respectively. In addition, the semiconductor device includes a gate dielectric layer between the first fin and the first gate electrode and between the second fin and the second gate electrode. Further, the semiconductor device includes a dummy gate electrode over the substrate, and the dummy gate electrode is between the first gate electrode and the second gate electrode. An upper portion of the dummy gate electrode is wider than a lower portion of the dummy gate electrode.