The system and methods described herein operate on a plurality of images that include multiple views of the same scene, typically from slightly different viewing angles and/or lighting conditions. One of the images is selected as a reference image. For each image ray in a non-reference image, the system and methods resample a local region from the non-reference image's space to the reference image's space. The resampling is performed multiple times, each time with a different surface orientation hypothesis. The system and methods run cross-correlation style correlators on the resampled images, evaluate correlation scores for each of the resampled images, and select the surface orientation hypothesis associated with the highest correlation score. The system and methods project a peak of the correlation surface back through a geometry model for the selected surface orientation hypothesis to determine a three-dimensional (ground) location for the image ray.