A semiconductor memory device, and a method of manufacturing the same, includes a gate stack including an interlayer insulating layers and conductive patterns alternately stacked in a vertical direction on a substrate, a channel structure passing through the gate stack and having an upper end protruding above the gate stack, a memory layer surrounding a sidewall of the channel structure, and a source layer formed on the gate stack. The channel structure includes a core insulating layer extending in a central region of the channel structure in the vertical direction, and a channel layer surrounding a sidewall of the core insulating layer, the channel layer formed to be lower in the vertical direction than the core insulating layer and the memory layer.