A tracking algorithm includes a tracking failure detection technique using a key points selection algorithm. As a result, the described system is able to work on an embedded device in real time, providing high quality vehicle detection and tracking, in some embodiments. A vision system detects and tracks vehicles from sequences of images taken from another moving vehicle in real-time on an embedded platform. The system can achieve real-time performance on an embedded platform in some embodiments, taking into account that modern boosting detectors are too slow for use in such a system. A tracker includes an algorithm for tracking and an algorithm for detection of tracking failures. The tracking algorithm is based on an optical flow calculation for key points selected based on the distribution of features from the last detection. The algorithm for detection of tracking failures is based on an estimation of low confidence detections.