An image processing apparatus including a unit configured to acquire a current image from an inputted video and a background model which comprises a background image and foreground/background classification information of visual elements; a unit configured to determine first similarity measures between visual elements in the current image and the visual elements in the background model; and a unit configured to classify the visual elements in the current image as the foreground or the background according to the current image, the background image in the background model and the first similarity measures. Wherein, the visual elements in the background model are the visual elements whose classification information is the background and which neighbour to corresponding portions of the visual elements in the current image. Accordingly, the accuracy of the foreground detection could be improved.