A system and method of detecting an interaction between a plurality of objects. The method comprises receiving tracking information for the plurality of objects in a scene; generating a plurality of frames, each of the plurality of frames comprising an activation for each of the plurality of objects and representing a relative spatial relationship between the plurality objects in the scene determined from the received tracking information, the frames encoding properties of the objects using properties of the corresponding activations; determining, using a trained neural network, features associated with the plurality of objects from the generated plurality of frames using the activations and the relative spatial relationship between the objects, the features representing changes in the relative spatial relationship between the objects over time relating to the interaction; and detecting time localization of the interaction in the plurality of frames using the determined features.