A semiconductor memory device includes memory cells, a source line, a word line, a bit line, and a driver circuit. The memory cells are formed on a semiconductor layer and have a charge accumulation layer and a control gate on the charge accumulation layer. The word line is connected to gate of the memory cell. The bit line is electrically connected to a drain of the memory cell. The source line is electrically connected to a source of the memory cell. The driver circuit varies potential of the semiconductor layer in conjunction with potential of the source line.