A method for detecting temporal pattern anomalies in a video stream includes detecting an object in a current frame of the video stream, generating a processed current frame that contains the detected object, generating a feature representation of the processed current frame, clustering the feature representation in one or more primary clusters in a clustering space of the primary class, generating an information vector of the feature representation, that includes information regarding the primary class, the sub-class and one or more external factors associated with the feature representation, clustering each information vector into one or more secondary clusters, and reporting a next frame as an anomaly when a corresponding information vector is positioned outside a secondary cluster of a feature presentation of a previous frame.