A semiconductor device includes first and second fin-shaped silicon layers on a substrate, each corresponding to the dimensions of a sidewall pattern around a dummy pattern. First and second pillar-shaped silicon layers reside on the first and second fin-shaped silicon layers, respectively. An n-type diffusion layer resides in an upper portion of the first fin-shaped silicon layer and in upper and lower portions of the first pillar-shaped silicon layer. A p-type diffusion layer resides in an upper portion of the second fin-shaped silicon layer and upper and lower portions of the second pillar-shaped silicon layer. First and second gate insulating films and metal gate electrodes are around the first and second pillar-shaped silicon layers, respectively. A metal gate line is connected to the first and second metal gate electrodes and extends in a direction perpendicular to the first and second fin-shaped silicon layers.