An image processing device includes an image processor that calculates a motion vector between image data of a target frame and image data of a reference frame in units of a block, and a reference frame image memory that retains image data of a past frame as the image data of the reference frame. Furthermore, the image processing device includes a primary memory that retains a matching processing range of the reference frame in calculation by the image processor, and a secondary memory that reads out and retains image data of a desired range from the image data of the reference frame stored in the reference frame image memory. The secondary memory reads out data of the matching processing range from the retained image data and supplies the read data to the primary memory.