A semiconductor memory device comprising a stacked unit, a semiconductor pillar, a charge storage layer, and a non-insulating film. The stacked unit includes first conductive layers and first insulating layers which are stacked alternately. The semiconductor pillar passes through the stacked body and the semiconductor pillar has a tubular structure. The charge storage layer is provided between the semiconductor pillar and each of the first conductive layers. The non-insulating film is provided inside the tubular structure and has a non-insulating member. The first effective impurity concentration of the non-insulating film is lower than a second effective impurity concentration of the semiconductor pillar.