A semiconductor memory device includes a plurality of wiring layers formed on a substrate, one or more first pillars penetrating through the wiring layers on a memory region of the substrate and in contact with the substrate, a plurality of memory transistors being formed at portions of each of the one or more first pillars that penetrate the wiring layers, and one or more second pillars penetrating through at least one of the wiring layers on a peripheral region of the substrate and in contact with the substrate. Each of the first and second pillars includes a semiconductor portion, a first insulating layer formed around the semiconductor portion, a charge accumulation layer formed around the first insulating layer, and a second insulating layer formed around the charge accumulation layer.