Described is providing an action model (classifier) for automatically detecting actions in video clips, in which unlabeled data of a target dataset is used to adaptively train the action model based upon similar actions in a labeled source dataset. The target dataset comprising unlabeled video data is processed into a background model. The action model is generated from the background model using a source dataset comprising labeled data for an action of interest. The action model is iteratively refined, generally by fixing a current instance of the action model and using the current instance of the action model to search for a set of detected regions (subvolumes), and then fixing the set of subvolumes and updating the current instance of the action model based upon the set of subvolumes, and so on, for a plurality of iterations.