In embodiments of image color and tone style transfer, a computing device implements an image style transfer algorithm to generate a modified image from an input image based on a color style and a tone style of a style image. A user can select the input image that includes color features, as well as select the style image that includes an example of the color style and the tone style to transfer to the input image. A chrominance transfer function can then be applied to transfer the color style to the input image, utilizing a covariance of an input image color of the input image to control modification of the input image color. A luminance transfer function can also be applied to transfer the tone style to the input image, utilizing a tone mapping curve based on a non-linear optimization to estimate luminance parameters of the tone mapping curve.