In an embodiment, a method comprises receiving a set of three dimensional (3D) points estimated from a plurality of images; selecting a subset of the 3D points that, as a group, provide at least a desired amount of constraint on each of a plurality of camera parameters; and performing non-linear optimization over the selected subset of 3D points to recover the plurality of camera parameters and the 3D points. The selected subset excludes one or more 3D points in the set.