An audio signal processing method includes: controlling the volume of a digital audio signal according to a latest value of an adaptive gain value which is adjustable; detecting whether data carried by the digital audio signal satisfies a specific criterion and utilizing a pendulum counter to count a number of times the data carried by the digital audio signal satisfies the specific criterion; if the counter value of the pendulum counter reaches an upper bond, selectively decreasing the adaptive gain value according to a decrement to decrease the volume of the digital audio signal; and if the counter value of the pendulum counter reaches a lower bond, selectively increasing the adaptive gain value according to an increment to increase the volume of the digital audio signal, where the magnitude of the decrement and the magnitude of the increment are different from each other.