An image processing apparatus acquires a plurality of input images which are serially photographed, detects a plurality of object regions from each input image, and associates object regions in a past input image with object regions in a current input image. If any object region that has not been appropriately associated exists in the object regions detected from the past input image, the apparatus calculates object region movement information for each size of the object region based on the object regions mutually associated between the past input image and the current input image, and estimates a position of the object region of the current input image that has not been associated based on the movement information for each size of the object region and a size of the object region that has not been associated.