An audio signal processing device includes a high pass filter to convert an input audio signal into a first audio signal and to output the first audio signal, a displacement estimation unit to estimate displacement amplitude of a speaker diaphragm when the input audio signal is inputted, a saturation processing unit to perform saturation processing to the displacement amplitude estimated in the displacement estimation unit or to a signal obtained by correcting the displacement amplitude, an audio signal generation unit to generate a second audio signal by using displacement amplitude obtained after the saturation processing in the saturation processing unit, and an output generation unit to generate an output signal by using the first and the second audio signals. With this configuration, a cracking sound of a speaker is reduced and a user is allowed to feel low frequency components.