Sub pixel image alignment includes mapping first pixels from a first image and second pixels from a second image to a coordinate system and applying one or more sub-pixel shifts to the mapped first pixels. For each sub-pixel shift, an overall energy is calculated and is based on a plurality of gradients that represent changes in a channel value among the shifted first pixels and the mapped second pixels. The sub-pixel alignment further includes determining the sub-pixel shift that provides the lowest overall energy.