A sound source localization system includes a plurality of microphones for receiving a signal as an input from a sound source; a time-difference extraction unit for decomposing the signal inputted through the plurality of microphones into time, frequency and amplitude using a sparse coding and then extracting a sparse interaural time difference (SITD) inputted through the plurality of microphones for each frequency; and a sound source localization unit for localizing the sound source using the SITDs. A sound source localization method includes receiving a signal as an input from a sound source; decomposing the signal into time, frequency and amplitude using a sparse coding; extracting an SITD for each frequency; and localizing the sound source using the SITDs.