In one embodiment, a semiconductor memory device includes a semiconductor substrate, and isolation layers formed in a surface of the semiconductor substrate, and separating the semiconductor substrate into active areas, the isolation layers and the active areas being alternately arranged along a predetermined direction parallel to the surface of the semiconductor substrate, a height of upper surfaces of the isolation layers being lower than a height of an upper surface of the semiconductor substrate. The device further includes diffusion layers formed on surfaces of the active areas, and a stress liner formed on upper surfaces and side surfaces of the diffusion layers, and formed of a material having a lattice constant smaller than a lattice constant of a material formed of the semiconductor substrate.