In one embodiment, objects are tracked by calculating affinity scores with multiple cues in cascading stages. In each stage, affinity scores are calculated for currently detected objects based on a cue. The affinity scores are then fed into subsequent stages to inform subsequent calculations based on subsequent cues, where calculations can be skipped and resources saved. Object tracks are updated every frame, to assist in autonomous vehicle driving.