A three-dimensional (3D) mapping system can be configured to generate a 3D map of a real-world environment using annotation of large image data sets, in which terrestrial imagery can be programmatically labeled with accurate labels using remotely sensed overhead image data. The 3D mapping system can implement photogrammetry to create a point cloud. Each pixel in the point cloud can be classified based on a consensus of each frame. The point cloud can be co-registered to a remotely sensed reference dataset to provide precise spatial coordinates for each pixel. Different patches of point clouds can be stitched together to provide a complete 3D map for a given area, such as a downtown area of a city.