A pillar-shaped semiconductor memory device includes Si pillars arranged in at least two rows; a tunnel insulating layer; a data charge storage insulating layer; first, second, and third interlayer insulating layers; and first and second conductor layers, all of which surround outer peripheries of the Si pillars, the first and second conductor layers being located at the same height in a perpendicular direction. A row of the semiconductor pillars is interposed between the first and second conductor layers of Si pillars arranged in an X direction. Shapes of the first and second conductor layers facing the semiconductor pillars are circular arcs. Adjacent circular arcs of the first conductor layer are in contact with each other, and adjacent circular arcs of the second conductor layer are in contact with each other. A pitch length of the Si pillars in the X direction is smaller than that in a Y direction.