According to an embodiment, a semiconductor memory device includes a stacked body in which insulating layers and electrode films are alternately stacked, a pillar member arranged in a memory hole that is disposed in the stacked body in a thickness direction, and a semiconductor layer provided below the pillar member. The pillar member has a structure in which a memory film and a channel layer are stacked in order from a side of the stacked body. The channel layer has a stacked structure that includes an outer channel semiconductor layer, an intermediate layer made of an insulating material, and an inner channel semiconductor layer, from a side of the memory film. Both of the outer channel semiconductor layer and the inner channel semiconductor layer are electrically connected to the semiconductor layer.