An image processing apparatus includes a determination unit, a search unit, a weight assignment unit and a filling unit. The determination unit determines whether a hole is surrounded by the foreground in a disparity map or a depth map. The search unit searches for multiple relative backgrounds along multiple directions when the hole is surrounded by the foreground. The weight assignment unit respectively assigns weights to the relative backgrounds. The filling unit selects an extremum from the weights, and fills the hole according to the relative background corresponding to the extremum.