A method and associated imaging system for classifying at least one concept type in a video segment is disclosed. The method associates an object concept type in the video segment with a spatio-temporal segment of the video segment. The method then associates a plurality of action concept types with the spatio-temporal segment, where each action concept type of the plurality of action concept types is associated with a subset of the spatio-temporal segment associated with the object concept type. The method then classifies the action concept types and the object concept types associated with the video segment using a conditional Markov random field (CRF) model where the CRF model is structured with the plurality of action concept types being independent and indirectly linked via a global concept type assigned to the video segment, and the object concept type is linked to the global concept type.