A semiconductor memory device according to the present embodiment includes a semiconductor substrate, a structure including a plurality of insulating films and a plurality of conductive films alternately stacked on the semiconductor substrate, and a pillar penetrating the structure. The plurality of conductive films include a plurality of first conductive films and a second conductive film arranged closer to the semiconductor substrate than the plurality of first conductive films. The pillar has a first epitaxial growth layer doped with boron and carbon in a part in contact with the semiconductor substrate, and configured to functions as a part of a source side select gate transistor together with the second conductive film. The plurality of first conductive films configured to functions as a part of a plurality of non-volatile memory cells.