A semiconductor memory device includes a semiconductor substrate, a first stacked body including a plurality of first insulating layers and a plurality of first conductive layers alternately stacked in a first direction intersecting a surface of the semiconductor substrate, a second stacked body including a plurality of second insulating layers and a plurality of second conductive layers alternately stacked in the first direction of the first stacked body, a third insulating layer arranged between the first stacked body and the second stacked body, and a pillar penetrating the first stacked body, the third insulating layer, and the second stacked body, the pillar comprising a semiconductor layer extending in the first direction and a charge storage layer extending in the first direction and arranged between the plurality of first conductive layers and the semiconductor layer and between the plurality of second conductive layers and the semiconductor layer.