A computer-implemented method, system, and computer program product are provided for activity recognition. The method includes receiving, by a processor, a plurality of videos, the plurality of videos including labeled videos and unlabeled videos. The method also includes extracting, by the processor with a feature extraction convolutional neural network (CNN), frame features for frames from each of the plurality of videos. The method additionally includes estimating, by the processor with a feature aggregation system, a vector representation for one of the plurality of videos responsive to the frame features. The method further includes classifying, by the processor, an activity from the vector representation. The method also includes controlling an operation of a processor-based machine to react in accordance with the activity.