A semiconductor memory device includes first and second wiring layers above a semiconductor substrate, a memory pillar extending through the first and second wiring layers, a first plug contacting the first wiring layer, a second plug contacting the second wiring layer, a first pillar adjacent to the first plug and extending through the first wiring layer, and a second pillar adjacent to the second plug and extending through the first and second wiring layers. The memory pillar includes a first semiconductor layer, a second semiconductor layer over the first semiconductor layer, and a third insulating layer, a charge storage layer, and a fourth insulating layer on a side surface of the second semiconductor layer. The distance between the center of the first plug and the center of the first pillar is greater than the distance between the center of the second plug and the center of the second pillar.