Provided is an image processing apparatus which includes a setting unit assigning a control block, which is a control unit of a filter process that is locally performed with respect to an image, to an initial position of the image determined based on a predetermined reference point; a movement unit moving the control block, which has been assigned to the initial position of the image by the setting unit, up to a position in which the result of the filter process is improved; and a filter processing unit performing the filter process for the respective control blocks which has been moved by the movement unit.