A semiconductor memory device includes a first stack including lower conductive patterns separated from each other and stacked on a substrate to form a lower stepped structure, a support pillar passing through the first stack and including an insulating layer, a second stack including upper conductive patterns separated from each other and stacked on the first stack, the upper conductive patterns including an upper stepped structure that does not overlap with the lower stepped structure and the support pillar, a channel structure passing through the second stack and the first stack, and a memory layer surrounding a sidewall of the channel structure.