Techniques are disclosed for identifying multimodal subevents within an event having spatially-related and temporally-related features. In one example, a system receives a Spatio-Temporal Graph (STG) comprising (1) a plurality of nodes, each node having a feature descriptor that describes a feature present in the event, (2) a plurality of spatial edges, each spatial edge describing a spatial relationship between two of the plurality of nodes, and (3) a plurality of temporal edges, each temporal edge describing a temporal relationship between two of the plurality of nodes. Furthermore, the STG comprises at least one of: (1) variable-length descriptors for the feature descriptors or (2) temporal edges that span multiple time steps for the event. A machine learning system processes the STG to identify the multimodal subevents for the event. In some examples, the machine learning system comprises stacked Spatio-Temporal Graph Convolutional Networks (STGCNs), each comprising a plurality of STGCN layers.