The disclosed embodiments provide a method and system for operating a portable electronic device. The portable electronic device includes a camera that obtains a set of images and one or more inertial sensors that obtain inertial data associated with the portable electronic device. The portable electronic device also includes an analysis apparatus that uses the set of images to obtain a set of features in proximity to the portable electronic device. Next, the analysis apparatus updates a set of locations of the features based on the set of images and the inertial data. Finally, the analysis apparatus uses the set of features and the set of locations to provide a model of an environment around the portable electronic device without requiring the use of specialized hardware to track the features and the locations.