A semiconductor memory device and a memory system having the same are provided. The semiconductor memory device includes a memory cell array including plural memory cell array blocks, and a refresh controller configured to control the memory cell array blocks to perform a normal refresh operation and a hammer refresh operation. The refresh controller controls one or more third memory cell array blocks excluding a first memory cell array block and one or more second memory cell array blocks adjacent to the first memory cell array block to perform the hammer refresh operation while the normal refresh operation is performed on the first memory cell array block among the memory cell array blocks.