An opt-keyframe reconstruction technique for selecting and reconstructing optimizing keyframes to provide a better reconstruction in a structure from motion (SFM) technique. The technique may, for example, be used in an adaptive reconstruction algorithm implemented by a general SFM technique. This technique may add and reconstruct optimizing frames to a set of keyframes already generated by an initialization technique and by an adaptive technique for iteratively selecting and reconstructing additional keyframes. In addition, the technique may determine and remove outlier points from the projection, and determine and recover inlier points in the projection. Adding the opt-keyframes and inlier points may result in additional, and possibly shorter, point trajectories being included in the reconstruction, thus providing a better reconstruction that may be more suited for later operations that may be applied to the image sequence.