Disclosed is a method and system for object detection and tracking. Spatio-temporal information for a foreground/background appearance module is updated, based on a new input image and the accumulated previous appearance information and foreground/background information module labeling information over time. Object detection is performed according to the new input image and the updated spatio-temporal information and transmitted previous information over time, based on the labeling result generated by the object detection. The information for the foreground/background appearance module is repeatedly updated until a convergent condition is reached. The produced labeling result from objection detection is considered as a new tracking measurement for further updating on a tracking prediction module. A final tracking result may be obtained through the updated tracking prediction module, which is determined by the current tracking measurement and the previous observed tracking results. The tracking object location at the next time is predicted. The returned predicted appearance information for the foreground/background object is used as the input for updating the foreground and background appearance module. The returned labeling information is used as the information over time for the object detection.