A semiconductor memory device includes a memory bank including a plurality of word lines, and a refresh operation control unit suitable for performing a first refresh operation for a first adjacent word line group of a target word line of the plurality of word lines, and performing a second refresh operation for a second adjacent word line group of the target word line after the first refresh operation, in response to a smart refresh command.