Aspects of the disclosure relate to detecting sidewalks adjacent to roads. In this regard, a set of potential sidewalk areas adjacent to one or more roads in a vehicle's vicinity may be determined based on map data. Topology data for the set of potential sidewalk areas may be generated based on sensor data received from a perception system of the vehicle. The set of potential sidewalks may be filtered to remove areas unlikely to include a sidewalk. The vehicle may be operated based on the filtered set of potential sidewalk areas, which may include taking precautionary measures when within a predetermined distance from any of the filtered set of potential sidewalks.