A semiconductor memory device according to an embodiment includes a first stacked body, a second stacked body, an intermediate conductive layer, an intermediate insulating layer, a semiconductor pillar, a charge storage film, and an insulating film. The semiconductor pillar includes a first part, a second part, and a third part. The charge storage film includes a first charge storage portion and a second charge storage portion. The charge storage film includes at least one first element selected from the group consisting of nitrogen, hafnium, and aluminum. The insulating film provides in at least a portion between the intermediate conductive layer and the first part. The insulating film not includes the first element, or the insulating film has a concentration of the first element lower than a concentration of the first element of the charge storage film.