According to embodiments, a semiconductor memory device includes a plurality of control gate electrodes laminated above a substrate and extend in a first direction and a second direction, and a memory pillar that has one end connected to the substrate, has longitudinally a third direction intersecting with the first direction and the second direction, and is opposed to the plurality of control gate electrodes. The memory pillar includes a core insulating layer and a semiconductor layer arranged around the core insulating layer. The semiconductor layer includes a first portion and a second portion positioned at a substrate side of the first portion. A width in the first direction or the second direction of the semiconductor layer at at least a part of the first portion is larger than a width in the first direction or the second direction of the second portion.