An audio signal processing apparatus includes a splitting unit for splitting an audio signal of a first system and another audio signal of a second system into pluralities of frequency band components, a level comparing unit for calculating a level ratio or a level difference between each of the frequency bands of the first system and each of the frequency bands of the second systems, and an output control unit for removing frequency band components whose level ratio or level difference calculated by the level comparing unit is equal and substantially equal to a predetermined value from at least one of the first and second systems.