A method, system, and computer-readable storage medium for automatic segmentation of a video sequence. A segmentation shape prediction and a segmentation color model are determined for a current image of a video sequence based on existing segmentation information for at least one previous image of the video sequence. A segmentation of the current image is automatically generated based on a weighted combination of the segmentation shape prediction and the segmentation color model. The segmentation of the current image is stored in a memory medium.