A semiconductor memory device includes a substrate, a plurality of first conductive layers, a second conductive layer, a first pillar, and a second pillar. The plurality of first conductive layers are stacked over the substrate in a first direction. The second conductive layer is disposed over the plurality of first conductive layers. The first pillar extends inside the plurality of first conductive layers in the first direction. The first pillar includes a first semiconductor portion including a first semiconductor of single-crystal. The second pillar extends inside the second conductive layer in the first direction. The second pillar includes an insulating portion serving as an axis including an insulator and a second semiconductor portion which is disposed on an outer circumference of the insulating portion in view of the first direction. The second semiconductor portion is in contact with the first semiconductor portion and includes a second semiconductor of poly-crystal.