A method and image processing system for removing a visual object from an image, e.g., a moving image, is described. An image is provided in which a visual object to be removed is selected, for which a mask is determined. Pixels outside the mask that are intended to be used for replacing pixels inside the mask are selected, based on a similarity measure, comprising an appearance similarity measure representing the visual similarity between a previously selected pixel for replacing the pixel to be replaced and the pixel to be selected, and a geometry similarity measure, representing the closeness of the pixel to be selected to the pixel among the pixels outside the mask that was previously used for replacing the pixel adjacent to the pixel to be replaced. The pixels inside the mask are replaced by copying the properties of the selected pixels to the respective pixels to be replaced.