Techniques for determining map data for localizing a vehicle in an environment are discussed herein. A vehicle can capture sensor data such as image data and can detect objects in the sensor data. Detected objects can include semantic object such as traffic lights, lane lines, street signs, poles, and the like. Rays based on a pose of the sensor and the detected objects can be determined. Intersections between the rays can be determined. The intersections can be utilized to determine candidate landmarks in the environment. The candidate landmarks can be utilized to determine landmarks and to eliminate other candidate landmarks. The map data can be determined based on the three-dimensional locations and utilized to localize vehicles.