Aspects of the disclosure provide for controlling an autonomous vehicle having an autonomous driving mode. For instance, sensor data generated by a perception system of a vehicle may be received. The sensor data identifying objects in an environment of the vehicle. A trajectory corresponding to a future path of the vehicle may be received. That the vehicle is approaching an unmarked crosswalk situation may be determined based on the trajectory and the sensor data. In response to the determination, an area for an unmarked crosswalk is identified. The vehicle may be controlled in the autonomous driving mode based on the determination and the area.