A system for calibrating stereo cameras using time-of-flight (ToF) includes a stereo camera system and a ToF sensor onboard a vehicle. The system receives stereo image data depicting a scene and ToF sensor data captured during stationary or moving operations of the vehicle. The system associates the ToF sensor data to the stereo image data by projection onto the stereo image data to filter out stereo depth errors. The system converts distance measurements based on the ToF sensor data to disparities in the stereo image data. The system further filters out the disparities between the ToF sensor and the stereo camera system using a voting scheme to populate a histogram with disparities. The system can select the peak of the histogram and correct the stereo image data with the disparity results in the peak, to correct stereo image data during operation of the vehicle.