A system tracks multiple moving objects in a stream of video data. In an embodiment, the system uses a combination of video motion detection data and color based data. In another embodiment, the system combines this data into a condensation framework. In yet another embodiment, the integration of the video motion data and the color-based data is done is real time. In further embodiments, the robustness of the tracking is enhanced by applying heuristics.