An apparatus for processing an encoded audio signal having a plurality of downmix signals associated with a plurality of input audio objects and object parameters. The apparatus includes a grouper configured to group the downmix signals into groups of downmix signals associated with a set of input audio objects. The apparatus includes a processor configured to perform at least one processing step individually on the object parameters of each set of input audio objects in order to provide group results. Further, there is a combiner configured to combine the group results or processed group results in order to provide a decoded audio signal. The invention also refers to a corresponding method.