Described herein are systems, methods, storage media, and computer programs for tracking a plurality of feature points in a first image and a second image. In one embodiment, a first layer of a firstimage pyramid for the first image and a first layer of a second image pyramid for the second image are obtained. Based on expected pixel locations of the plurality of feature points in the first layerof the second image pyramid, a feature point is selected from the plurality of feature points. For the feature point, a first pixel in the first layer of the first image pyramid is determined, and asecond pixel in the first layer of the second image pyramid is determined, where the first pixel substantially matches the second pixel.