A semiconductor memory device includes: a plurality of first channel columns including a plurality of first channel layers that are arranged in a direction and offset by their centers; a plurality of second channel columns alternately arranged with the plurality of first channel columns and having a plurality of second channel layers that are arranged in the direction and offset by their centers; first insulating layers and first conductive layers alternately stacked to surround the first channel layers; second insulating layers and second conductive layers stacked to surround the second channel layers; and spacers placed between the first channel columns and the second channel columns and interposed between the first conductive layers and the second conductive layers.