According to one embodiment, a semiconductor memory device includes a stacked body of first conductor layers and second conductor layers. A pillar including a semiconductor layer extends along through the stacked body in a first direction. A charge storage layer is between the conductor layers and the semiconductor layer. The semiconductor layer includes a first portion extending along the first direction from an uppermost first conductor layer to a lowermost second conductor layer and a second portion above the first portion in the first direction. The second portion has a diameter that decreases with increasing distance along the first direction from the first portion.