Embodiments are disclosed for reconstructing freeform gradients from an input image. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving an input image, computing an outline of the input image, identifying a set of candidate color handles for the input image, each candidate color handle of the set of candidate color handles representing an extremum point for a color in the input image, generating a reconstructed image using a subset of the set of candidate color handles, determining a reconstruction error by computing a difference between the input image and the reconstructed image, and providing the reconstructed image when the reconstruction error is below a threshold value.