Probabilistic determination of selected image portions is described. In one or more implementations, a selection input is received for selecting a portion of an image. For pixels of the image that correspond to the selection input, probabilities are determined that the pixels are intended to be included as part of a selected portion of the image. In particular, the probability that a given pixel is intended to be included as part of the selected portion of the image is determined as a function of position relative to center pixels of the selection input as well as a difference in one or more visual characteristics with the center pixels. The determined probabilities can then be used to segment the selected portion of the image from a remainder of the image. Based on the segmentation of the selected portion from the remainder of the image, selected portion data can be generated that defines the selected portion of the image.