A technique is disclosed for determining a group in which to add a new image set. The new image set is captured by one or more cameras in a video surveillance system. Similarity scores are generated between the new image set and one or more image sets of a plurality of groups, wherein each group includes one or more image sets of at least one object. The new image set is added to a group based on one or more factors. Also, a technique is disclosed for determining when to close a group, wherein a closed group is one to which an image set may not be added.