According to an embodiment, a semiconductor memory device comprises: a stacked body that includes a plurality of control gate electrodes stacked above a substrate; a memory columnar body that extends in a first direction above the substrate and configures a memory string along with the stacked body; and a source contact that extends in the first direction and is electrically connected to one end of the memory string. Moreover, this source contact is adjacent to the stacked body via a spacer insulating layer. Furthermore, a spacer protective layer including a nitride or a metal oxide is provided between these source contact and spacer insulating layer.