A semiconductor memory device and a method of forming the same, the semiconductor memory device includes a substrate, a plurality of bit lines, a gate, a spacer layer and a first spacer. The substrate has a memory cell region and a periphery region, the a plurality of bit lines are disposed on the substrate, within the memory cell region, and the gate is disposed on the substrate, within the periphery. The spacer layer covers the bit lines and a sidewall of the gate. The first spacer is disposed at two sides of the gate, covers on the spacer layer.