An apparatus for processing an audio signal and method thereof are disclosed. The present invention includes receiving a downmix signal and side information; extracting extension type identifier indicating whether extension area includes a residual signal from the side information; when the extension type identifier indicates that the extension area includes the residual signal, extracting control restriction information for residual using mode from the side information; receiving control information for controlling gain or panning of at least one object signal; estimating modified control information based on the control information and the control restriction information; obtaining at least one of enhanced object signal and one or more regular object signals from the downmix signal using the residual signal; and, generating an output signal using the modified control information and at least one of enhanced object signal and one or more regular object signal, wherein the control restriction information for residual using mode relates to a parameter indicating limiting degree of the control information in case of the residual using mode.