According to one embodiment, a semiconductor memory device includes a semiconductor substrate, a plurality of element-separating insulators, and contacts. The plurality of element-separating insulators partition the upper layer portion into a plurality of active areas extending in a first direction. The contacts are connected to the active areas. A recess is made in a part in the first direction of an upper surface of each of the active areas. The recess is made across the entire active area in a second direction orthogonal to the first direction. Positions in the first direction of two of the contacts connected respectively to mutually-adjacent active areas are different from each other. One of the contacts is in contact with a side surface of the recess and not in contact with a bottom surface of the recess.