A semiconductor device includes a pillar-shaped semiconductor layer formed on a substrate; a first insulator surrounding the pillar-shaped semiconductor layer; a first gate surrounding the first insulator and made of a metal having a first work function; a second gate surrounding the first insulator and made of a metal having a second work function different from the first work function, the second gate being located below the first gate; a third gate surrounding the first insulator and made of a metal having the first work function, the third gate being located below the second gate; and a fourth gate surrounding the first insulator and made of a metal having the second work function different from the first work function, the fourth gate being located below the third gate. The first gate, the second gate, the third gate, and the fourth gate are electrically connected together.