Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for group identification and monitoring. The methods, systems, and apparatus include actions of determining, using one or more first frames of a video sequence, that two people each of whom is depicted in at least one of the first frames satisfy a grouping criteria that indicates that the two people should be grouped for video analysis, determining, using one or more second frames of the video sequence at least some of which were captured after the one or more first frames and which depict at least one of the two people, that the two people satisfy a degrouping criteria that indicates that the two people should not be grouped for video analysis, determining that a physical distance between the two people violates a distance criteria, and providing an alert that the distance criteria is violated.