A semiconductor device includes a first pillar-shaped silicon layer formed on a planar silicon layer, a gate insulating film formed around the first pillar-shaped silicon layer, a first gate electrode formed around the gate insulating film, a gate line connected to the first gate electrode, a first first-conductivity-type diffusion layer formed in an upper portion of the first pillar-shaped silicon layer, a second first-conductivity-type diffusion layer formed in a lower portion of the first pillar-shaped silicon layer and an upper portion of the planar silicon layer, a first sidewall having a laminated structure of an insulating film and polysilicon and being formed on an upper sidewall of the first pillar-shaped silicon layer and an upper portion of the first gate electrode, and a first contact formed on the first first-conductivity-type diffusion layer and the first sidewall.