A vision system of a vehicle includes a camera configured to be disposed at a vehicle so as to have a field of view exterior of the vehicle. An image processor is operable to process image data captured by the camera. The image processor is operable to determine lane markers on a road on which the vehicle is traveling. The image processor processes intensity gradient information of captured image data to determine lane markers, and, responsive to processing of captured image data, the image processor is operable to detect straight or curved lane markers. The image processor is operable to adapt the processing of lane marker image data in subsequent frames of captured image data responsive to image processing of lane marker image data in previous frames of captured image data.