A system includes sensors and a tracking subsystem. The subsystem receives frames of top-view images generated by the sensors. The subsystem tracks a first, second, and third object, based on received frames. The subsystem detects that the first object is within a threshold distance of the second object. In response, the subsystem determines a probability that the first object switched identifiers with the second object and updates candidate lists accordingly for the first and second objects. The updated first candidate list includes a probability that the first object is associated with a first identifier and a probability that the first object is associated with a second identifier. The updated second candidate list includes a probability that the second object is associated with the first identifier and a probability that the second object is associated with the second identifier.