A processing device includes a storage device to store data and a processor, operably coupled to the storage device, the processor to receive a token stream comprising a plurality of tokens generated based on a byte stream comprising a plurality of bytes, wherein each token in the token stream comprises at least one symbol associated with a respective byte in the byte stream, and wherein the at least one symbol represents one of the respective byte, a length of a first byte string starting from the respective byte, or a byte distance between the first byte string and a matching second byte string, generate a graph comprising a plurality of nodes and edges based on the token stream, wherein each token in the token stream is associated with a respective node connected by at least one edge to another node, and wherein the at least one edge is associated with a cost function to encode the at least one symbol stored in the each token, identify, based on the graph, a path between a first node associated with a beginning token of the token stream and an end node associated with a last token of the token stream, wherein the path comprises a subset of nodes and edges linking the subset of nodes, and perform variable-length encoding of a subset of tokens associated with the subset of nodes to generate an output data.