A semiconductor memory device includes: an input control circuit suitable for providing an active address which is input together with an active command, as an input address; a plurality of latches suitable for sequentially storing, as a latch address, the input address according to input control signals and outputting the latch addresses as a target address according to output control signals; a plurality of counters respectively corresponding to the latches and each suitable for increasing, when the active address matches the latch address stored in the latch, a counting value corresponding to the latch; and a refresh controller suitable for dividing the counters and the latches into a plurality of groups based on the counting values and generating, in response to a refresh command, reset signals for initializing the counters included in one group of the groups.