The disclosure relates to an operating method of controller, and memory system having the same, the method controls a semiconductor memory device including a plurality of memory blocks. The method includes: receiving read data output according to a first read operation performed on a selected memory block; selecting a read voltage set group from a read retry table based on a read error related indicator providing an indication that an error correction failure has occurred; and selecting a read voltage set from the selected read voltage set group based on whether the read error related indicator is greater than or equal to a predetermined reference value. The selected read voltage set has a minimum average distance with respect to a read voltage set used for the first read operation, and has a minimum first read voltage distance with respect to the read voltage set used for the first read operation.