An audio signal processing apparatus for outputting an input digital audio signal as a positive phase PWM signal and a negative phase PWM signal. The resolution of the input digital audio signal is converted to a low resolution and noise shaping is performed to generate a positive phase digital signal (REF_P) and a negative phase digital signal (REF_N). Both signal patterns are compared at a buffer and pattern comparator section when muting. If the signal patterns match, a selector outputs a set value having 50% duty instead of the digital signals to optimize timing for switching to 50% duty when muting.