Described is a system for predicting multi-agent movements. A Radon Cumulative Distribution Transform (Radon-CDT) is applied to pairs of signature-formations representing agent movements. Canonical correlation analysis (CCA) components are identified for the pairs of signature-formations. Then, a relationship between the pairs of signature formations is learned using the CCA components. A counter signature-formation for a new dataset is predicted using the learned relationship and a new signature-formation. Control parameters of a device can be adjusted based on the predicted counter signature-formation.