Multi-scale processing may be used to reduce the memory and computational requirements of optimization algorithms for image labeling, for example, for object segmentation, 3D reconstruction, stereo correspondence, optical flow and other applications. For example, in order to label a large image (or 3D volume) a multi-scale process first solves the problem at a low resolution, obtaining a coarse labeling of an original high resolution problem. This labeling is refined by solving another optimization on a subset of the image elements. In examples, an energy function for a coarse level version of an input image is formed directly from an energy function of the input image. In examples, the subset of image elements may be selected using a measure of confidence in the labeling.