Disclosed is an audio signal processing device. The audio signal processing device includes a receiving end configured to receive a 2-channel stereo signal and a processor configured to process the 2-channel stereo signal. The processor is configured to filter the 2-channel stereo signal using a spatial distortion removal filter and output the filtered 2-channel stereo signal to a speaker including two or more channels. The spatial distortion removal filter is a filter for offsetting distortion that occurs when the output signal is transmitted from the speaker to a listener, and includes an ipsilateral filter applied to an ipsilateral signal of the 2-channel audio signal and a contralateral filter applied to a contralateral signal of the 2-channel audio signal.