An image decoding apparatus according to an embodiment includes a decoding section configured to encode an encoded stream and output decoded picture; and a control section configured to control decoding of the decoding section, and perform memory management of a memory section that has memory areas for data of four frames that stores the reference images of two frames, the decoded picture from the decoding section, and the decoded picture used for display, wherein the control section causes the decoding section to decode a number of pictures that is in accordance with a speed multiplying number in one frame period at a time of multiple speed reproduction, and exchanges data stored among the memory areas for data of four frames so that the decoded picture used for the display at a time of multiple speed reproduction is held in the memory area.