Systems and methods for computationally efficient and precise transformation recovery using two image frames are provided. A key image frame and a tracked image frame are selected from a series of image frames, a plurality of points on the key image frame are selected, and the plurality of points are then projected onto the tracked image frame to produce a number of corresponding points. Then, for each corresponding point, a patch of pixels is defined centered on the corresponding point. Each pixel of the patch is reprojected back onto the key image frame to identify a reprojected pixel in that image frame. An intensity value for each reprojected pixel is determined and each patch is parameterized. A normal equation for each patch is determined and then solved for the transformation.