A semiconductor device includes a first fin-shaped semiconductor layer on a semiconductor substrate, a first insulating film around the first fin-shaped semiconductor layer, a first pillar-shaped semiconductor layer on the first fin-shaped semiconductor layer, a first gate insulating film around the first pillar-shaped semiconductor layer, a first gate line formed around the first gate insulating film and extending in a direction perpendicular to the first fin-shaped semiconductor layer, a second diffusion layer disposed in a lower portion of the first pillar-shaped semiconductor layer, a third gate insulating film surrounding an upper portion of the first pillar-shaped semiconductor layer, a first contact electrode surrounding the third gate insulating film, a second contact electrode that connects an upper portion of the first contact electrode to an upper portion of the first pillar-shaped semiconductor layer, and a first magnetic tunnel junction memory element on the second contact electrode.