An image decoding apparatus, which receives a compressed image stream as an input signal and decodes the input signal, includes decoders for decoding the input signal by decoding predetermined data units. A task controller divides a decoding process for the input signal into process steps and allocates a data unit to be processed by each decoder and for each process step. Each process step has a dependency on an order of processing. The task controller detects a processing status of the decoder, and allocates a data unit and a process step to be processed to the decoder, depending on the detected processing status, a dependency between the process steps, and a dependency between frames included in the image stream.