The present invention discloses a semiconductor memory device and a forming method thereof. The semiconductor memory device includes a substrate, a plurality of bit lines, a strip-shaped isolation structure, a conductive residue, a plurality of columnar isolation structures and a plurality of conductive plugs. The bit lines are located on the substrate and extend along the first direction. The strip-shaped isolation structure is located at the ends of the bit lines and extends along the second direction, and the strip-shaped isolation structure includes a seam. In particular, the conductive residue is disposed in the seam. The columnar isolation structures are separated from each other and disposed between the bit lines. The conductive plugs are separated from each other and disposed between the bit lines, in which the conductive plugs and the conductive residue include the same conductive material.