Certain embodiments involve determining a diverse and representative subset of swatches of an input image. For example, a graphic manipulation application determine a salient object in an image. The graphic manipulation application extracts multiple swatches from the image. In some cases, the swatches include pixels included in the salient object. The graphic manipulation application computes selection scores for the multiple swatches by combining, for each swatch, a likelihood score indicating a representativeness of a pattern depicted by a respective swatch, and a dominance score indicating a dominance of the pattern depicted by the respective swatch. The graphic manipulation application generates, based on the selection scores of the multiple swatches, a subset of the multiple swatches extracted from the image.