A vehicle detecting system includes image taking means for taking an image of the surroundings of a subject vehicle, vehicle detecting means for detecting a vehicle from the taken image, and lateral-position detecting means for detecting a lateral position of the vehicle in the taken image. The lateral-position detecting means vertically divides, into a plurality of sections, a region set in a portion of the taken image in which an image of the vehicle is included or is presumed to be included. The lateral-position detecting means calculates average brightnesses of pixels belonging to pixel columns vertically extending in the sections and having a width corresponding to one pixel or a plurality of number of pixels. The lateral-position detecting means calculates differences in average brightness between pixel columns adjacent in a lateral direction, and detects the lateral position of the vehicle on the basis of absolute values of the differences.