User input-based object selection using multiple visual cues is described. User selection input is received for selecting a portion of an image. Once the user selection input is received, one of a plurality of visual cues that convey different information about content depicted in the image is selected for each pixel. The one visual cue is selected as a basis for identifying the pixel as part of the selected portion of the image or part of an unselected remainder of the image. The visual cues are selected by determining confidences, based in part on the user selection input, that the plurality of visual cues can be used to discriminate whether the pixel is part of the selected portion or part of the remainder. The information conveyed by the selected visual cues is used to identify the pixels as part of the selected portion or part of the remainder.