A method of forming a semiconductor memory device includes: forming a stack structure on a substrate, the stack structure including a plurality of dielectric layers and a plurality of sacrificial layers alternatingly stacked in a Z direction substantially perpendicular to the substrate; forming a plurality of source/drain trenches in the stack structure; conformally forming a barrier layer in the source/drain trenches, and then filling the source/drain trenches with a plurality of sacrificial segments; forming a protection layer over the stack structure to cover the barrier layer and the sacrificial segments; removing the sacrificial layers of the stack structure to form a plurality of spaces among the dielectric layers; forming a plurality of conductive layers in the spaces; sequentially removing the protection layer, the sacrificial segments and the barrier layer; and forming a plurality of memory structures in the source/drain trenches.