Techniques are disclosed for automatically transferring a style of at least two reference images to an input image. The resulting transformation of the input image matches the visual styles of the reference images without changing the identity of the subject of the input image. Each image is decomposed into levels of detail with corresponding energy levels and a residual. A style transfer operation is performed at each energy level and residual using the reference image that most closely matches the input image at each energy level. The transformations of each level of detail and the residual of the input image are aggregated to generate an output image having the styles of the reference images. In some cases, the transformations are performed on the foreground of the input image, and the background can be transformed by an amount that is proportional to the aggregated transformations of the foreground.