A computer-implemented method, system, and computer program product are provided for activity recognition in a surveillance system. The method includes receiving a plurality of unlabeled videos from one or more cameras. The method also includes classifying an activity in each of the plurality of unlabeled videos. The method additionally includes controlling an operation of a processor-based machine to react in accordance with the activity.