Embodiments related to using a foundational model for network packet traces. A technique includes receiving network traffic of a network and extracting features from the network traffic, the features having a function related to communications in the network. The technique includes generating tokens from the features, each of the features corresponding to a respective one of the tokens, training a machine learning model by inputting the tokens, the machine learning model being trained to output contextual embeddings for the tokens, and using the contextual embeddings to determine an anomaly in the network traffic.