The present disclosure provides a audio signal processing apparatus including, an amplitude detector configured to detect a noise start point of an audio signal including a noise signal by comparing an amplitude value of the audio signal with a threshold value, a frequency feature calculator configured to calculate a frequency feature representing at least a frequency characteristic of the audio signal after the noise start point, and a noise determiner configured to determine a leg continuously including high-frequency components equal to or higher than a reference frequency in the audio signal after the noise start point as a noise leg based on the frequency feature.