Systems and methods of segmenting images are disclosed herein. The similarity of images in a set of images is compared. A group of images is selected from the set of images. The images in the group of images are selected based on compared similarities among the images. An informative image is selected from the group of images. User-defined semantic information of the informative image is received. The group of images as a graph is modeled as a graph. Each image in the group of images denotes a node in the graph. Edges of the graph denote a foreground relationship between images or a background relationship between images. One or more images in the group of images are automatically segmented by propagating the semantic information of the informative image to images in the group of images having a corresponding graph node that is related to a graph node corresponding to the informative image. Segmentation results can be refined according to user provided image semantics.