According to one embodiment, a semiconductor memory device includes: semiconductor member; electrode member; charge accumulation member; a memory unit; and a control unit. Memory cell is formed at each crossing portion of the semiconductor member and the electrode member. The memory unit retains information indicating that the memory cell belongs to first group or second group. The control unit performs first step and second step, when reducing the charge accumulated in the charge accumulation member. In the first step, first voltage is applied both between the semiconductor member and the electrode member of the first group and between the semiconductor member and the electrode member of the second group. In the second step, second voltage is applied between the semiconductor member and the electrode member constituting the memory cell belonging to the second group.