Some embodiments provide a program that performs a color balance operation. The program identifies an image that includes several pixels. Each pixel includes a luma component value and chroma component values. The program analyzes the luma component values of the pixels in the image to identify several luma ranges. The program determines, for each luma range in the several luma ranges, a set of transforms for modifying chroma component values of pixels in the image in order to remove a color cast from the image. The program applies the sets of transforms to the image to remove the color casts from the image.