According to one embodiment, a semiconductor memory device includes a memory cell array, and first to third latch circuits. The first latch circuits hold information as to whether an associated column is defective. A pointer is set in the second latch circuits. The third latch circuits hold write data or read data. One of the third latch circuits is activated at a time the pointer is set to an associated second latch circuit when an associated first latch circuit holds the information indicating that the associated column is not defective. The pointer is sequentially shifted among the second latch circuits in synchronization with a clock. In shifting the pointer, the pointer skips one of the second latch circuits associated with one of the first latch circuit which holds the information indicating that the associated column is defective.