The system obtains a multimodal dataset containing different types of data gathered during operation of the computer system, wherein the multimodal dataset includes time-series data for different variables associated with operation of the computer system. Next, the system forms a set of feature groups from the multimodal dataset, wherein each feature group comprises variables from the multimodal dataset containing the same type of data. The system then computes a tripoint similarity matrix for each feature group, and aggregates the tripoint similarity matrices for the feature groups to produce a crossmodal tripoint similarity matrix. Next, the system uses the crossmodal tripoint similarity matrix to cluster the multimodal dataset to form a model. The system then performs prognostic-surveillance operations on real-time multimodal data received from the computer system, wherein the prognostic-surveillance operations use the model as a classifier to detect anomalies. When an anomaly is detected, the system triggers an alert.