According to one embodiment, a semiconductor memory device includes a semiconductor layer, a plurality of conductive layers, a plurality of insulating layers, an intermediate layer, and a controller. The conductive layers and the insulating layers are alternately provided. The intermediate layer is provided between the plurality of conductive layers and the semiconductor layer. The controller is configured to perform first and second operations. In first operation, the controller applies a first voltage to the semiconductor layer, applies a second voltage higher than the first voltage to a first conductive layer, and applies a third voltage to other conductive layers. In the second operation, the controller applies a fourth voltage to the semiconductor layer, applies a fifth voltage to the first conductive layer, and applies a sixth voltage to the other conductive layers.