An image processing device includes: a holder configured to hold a reference picture necessary for decoding of an access unit as a processing subject in decoding of a stream composed of access units including a header; an analyzer configured to analyze the stream to thereby detect the header for each of the access units; and a generator configured to, if a header indicating change of deletion order of pictures is detected as the header detected by the analyzer, delete a picture included in an access unit corresponding to the header in the deletion order indicated by the header and generate hold information indicating a reference picture to be held by the holder.