A semiconductor memory device includes an isolation layer formed in a substrate and defining an active region, a trench formed in the substrate and defining a part of the active region as an active pillar; a word line formed inside the trench, a sub-source line formed under the trench and crossing the word line, a main source line formed over the substrate, coupled to the sub-source line, and crossing the word line, a variable resistor pattern formed over the active pillar, and a bit line contacting the variable resistor pattern and crossing the word line.