The present invention relates to the method for updating scene model and video surveillance. A method is provided for updating a scene model in a video which is composed of a plurality of visual elements, comprising: a classifying step for classifying the visual elements in a scene into stationary visual elements and moving visual elements according to their appearance change rates; a border determining step for determining borders from the scene according to a spatial distribution information of the stationary visual elements and the moving visual elements; and an updating step for updating the scene model according to the determined borders in said scene.