A semiconductor memory device includes a substrate, an active structure, a shallow trench isolation and a plurality of word lines. The active structure is disposed in the substrate, and includes a plurality of first active fragments and a plurality of second active fragments extended parallel to each other along a first direction and the second active fragments are disposed outside a periphery of all of the first active fragments. The shallow trench isolation is disposed in the substrate to surround the active structure, and which includes a plurality of first portions and a plurality of second portions. The word lines are disposed in the substrate, parallel with each other to extend along a second direction, wherein at least one of the word lines are only intersected with the second active fragments, or at least one of the word lines does not pass through any one of the second portions.