A stereo vision based collision avoidance systems having stereo cameras that produce imagery that is processed to produce a depth map of a scene. A potential threat is detected in the depth map. The size, position, and velocity of the detected potential threat are then estimated, and a trajectory analysis of the detected potential threat is determined using the estimated position and the estimated velocity. A collision prediction based on the trajectory analysis is determined, and then a determination is made as to whether a collision is imminent based on the collision prediction and on the estimated size of the potential threat.