A semiconductor device includes first and second fin-shaped semiconductor layers on a substrate. First and second pillar-shaped semiconductor layers reside on the first and second fin-shaped semiconductor layers, respectively, where a width of the bottom of the first and second pillar-shaped semiconductors is equal to a width of the top of the first and second fin-shaped semiconductor layers, respectively. A gate insulating film and metal gate electrode are around underlying gate insulating layers on each fin-shaped semiconductor layer. A metal gate line is connected to the metal gate electrodes and extends in a direction perpendicular to the first and second fin-shaped semiconductor layers. Contacts reside on the upper portion of diffusion layers in upper portions of the first and second pillar-shaped semiconductor layers and are directly connected to the diffusion layers.