Systems, methods, and computer-readable storage media for automatically detecting skin tones in an input image are disclosed. An initial skin tone mask for the image may be created dependent on a general skin tone model. An upper threshold and/or a lower threshold may be applied to the initial skin tone mask to identify pixels most likely to be skin pixels and least likely to be skin pixels, respectively. These pixels may be used to produce an image-specific skin tone model, including one or more truncated Gaussian models for skin pixels and/or non-skin pixels defined in a three-dimensional color space. The image-specific skin tone model may be applied to the image to generate a final skin tone mask. Skin tone detection may be automatically or selectively performed in conjunction with image editing or image feature identification operations to target or exclude skin pixels or non-skin pixels during execution of the operations.