A method of forming a semiconductor memory device includes the following steps. First of all, a substrate is provided, and a plurality of gates is formed in the substrate, along a first direction. Next, a semiconductor layer is formed on the substrate, covering the gates, and a plug is then in the semiconductor layer, between two of the gates. Then, a deposition process is performed to from a stacked structure on the semiconductor layer. Finally, the stacked structure is patterned to form a plurality of bit lines, with one of the bit lines directly in contact with the plug.