A method of decoding an audio signal is disclosed, The present invention includes the steps of receiving the audio signal having a plurality of channel signals including an ambient component signal and a source component signal, extracting the ambient component signal and the source component signal of each of the channels based on correlation between the channel signals, modifying the ambient component signal using surround effect information, and generating the audio signal including a plurality of channels using the modified ambient component signal and the source component signal. Accordingly, in an apparatus for decoding an audio signal and method thereof according to the present invention, an ambient component signal is extracted and modified based on correlation and the modified ambient and source component signals are outputted using different signal output units, respectively. Therefore, the present invention enhances a stereo effect of the audio signal. And, a signal output unit for outputting a ambient component signal is arranged to have an output direction different from that of another signal output unit for outputting a source component signal, whereby a listener can be provided with an audio signal of which ambient sound is enhanced.