According to one embodiment, a semiconductor memory device includes a stacked body and a column. The stacked body includes a plurality of electrode layers. The column includes a semiconductor channel, a charge storage film, and a doped silicon layer. The semiconductor channel extends in the stacking direction. The semiconductor channel is a polycrystalline. An average grain size of crystals in a polycrystalline is not less than a film thickness of the semiconductor channel. The charge storage film is provided between the semiconductor channel and the electrode layers. The doped silicon layer contains a metal element and an impurity other than a metal element. The doped silicon layer is in contact with a top end of the semiconductor channel.