An image enlarging apparatus includes: a luminance data calculation unit configured to set a reference frame from a video image, and to calculate first luminance data for a high resolution video image; a position calculation unit configured to set target pixels in at least one of frames included in the video image except the reference frame, and to calculate corresponding positions to the target pixels in the reference frame with decimal accuracy; a correction amount calculation unit configured to parallelly calculate correction amounts of the first luminance data for the corresponding positions based on the first luminance data, third luminance data of the target pixels, and the corresponding positions; an addition unit configured to calculate sum of correction amounts from the correction amounts; and a luminance data correction unit configured to correct the first luminance data based on the sum of the correction amounts.