A system for generating one or more audio output signals is provided, having a decomposition module, a signal processor, and an output interface. The signal processor is configured to receive the direct component signal, the diffuse component signal and direction information. Moreover, the signal processor is configured to generate one or more processed diffuse signals depending on the diffuse component signal. For each audio output signal of the one or more audio output signals, the signal processor is configured to determine a direct gain, the signal processor is configured to apply the direct gain on the direct component signal to obtain a processed direct signal, and the signal processor is configured to combine the processed direct signal and one of the one or more processed diffuse signals to generate the audio output signal. The signal processor further has a gain function computation module and a signal modifier.