A ball detection and tracking system including one or more visual sensors and a detection and tracking agent that ranks a plurality of blob detection algorithms based on a detection metric and uses a selected base detection algorithm to identify one or more candidate blobs. Based on this, the agent is able to generate a track for the candidate blobs and assign one or more subsequent candidate blobs to a best ranked one of the tracks if the assignment satisfies a cost threshold.