A vision system of a vehicle includes a camera and an image processor. The camera is configured to be disposed at a vehicle so as to have a field of view exterior of the vehicle. The image processor is operable to process image data captured by the camera to classify patterns of retroreflective reflectors present in the field of view of the camera. The image processor compares determined patterns of retroreflective reflectors to a database of patterns and classifies patterns of retroreflective reflectors at least in part responsive to determination that determined patterns of retroreflective reflectors generally match a pattern of the database. The image processor may compare movement of determined patterns of retroreflective reflectors and pattern movements of the database over multiple frames of captured image data and may classify the retroreflective reflectors at least in part responsive to determination that the movements generally match.