A semiconductor memory device performs a refresh operation sequentially for a word line selected based on a row address when receiving a refresh request, and comprises: a memory cell array divided into M banks; a refresh counter for sequentially outputting a count value corresponding to the word line to be refreshed in response to the refresh request; and a row address converter for supplying row addresses different from one anther in at lest two banks among the M banks by converting the count value. In the semiconductor memory device, a predetermined number of selected word lines are refreshed at the same time in the banks in accordance with different patterns from one another, and the maximum value of the total number of the selected word lines refreshed at the same time for all the M banks is controlled to be lower than 2M.