The disclosure provides a semiconductor memory device including a substrate having a memory cell region and an alignment mark region; a dielectric layer covering the memory cell region and the alignment mark region; conductive vias in the dielectric layer within the memory cell region; an alignment mark trench in the dielectric layer within the alignment mark region; and storage structures disposed on the conductive vias, respectively. Each of the storage structures includes a bottom electrode defined from a bottom electrode metal layer, a magnetic tunnel junction (MTJ) structure defined from an MTJ layer, and a top electrode. A residual metal stack is left in the alignment mark trench. The residual metal stack includes a portion of the bottom electrode metal layer and a portion of the MTJ layer.