llmzip, compression that predicts the next token
A lossless compressor that replaces the dictionary with a language model. Every token costs only its surprise, the -log2 P(token) the model assigns it, arithmetic-coded to within a fraction of a bit of that. A 124M-parameter GPT-2 already beats gzip by 2.8x. A 0.5B model gets to 9.4x.
The idea
Compression and prediction are the same problem wearing different clothes. Shannon
fixed the exchange rate a long time ago: a symbol your model expected with
probability p is worth exactly -log2 p bits, and no coder
can do better on average. Classical compressors carry a weak model, a sliding window
of text they have already seen, and spend most of their bits on the parts of the
language they cannot predict.
A language model is a very good model of exactly that. So llmzip runs the model over the text one token at a time, takes the full probability distribution it produces for the next token, and arithmetic-codes the token that actually came next against that distribution. Predictable text costs almost nothing. Only the surprises cost bits. Decompression replays the identical loop: same model, same context, same distribution, so the decoder can invert every step.
The numbers
Measured on a 32,251-byte prefix of enwik9, the Hutter Prize corpus, so the ratios are directly comparable across methods. Ratio is raw size over compressed size, and bits/char is the same number from the other side, where plain text is 8 bits/char.
| Method | Size | Ratio | Bits/char |
|---|---|---|---|
| llmzip · Qwen2.5-0.5B | 3,424 B | 9.42x | 0.849 |
| llmzip · SmolLM2-135M | 3,544 B | 9.10x | 0.879 |
| llmzip · gpt2-medium | 3,971 B | 8.12x | 0.985 |
| llmzip · gpt2 (124M) | 4,467 B | 7.22x | 1.108 |
| llmzip · distilgpt2 | 5,233 B | 6.16x | 1.298 |
| brotli -11 | 9,891 B | 3.26x | 2.454 |
| bzip2 -9 | 11,239 B | 2.87x | 2.788 |
| lzma -9e | 11,696 B | 2.76x | 2.901 |
| gzip -9 | 12,307 B | 2.62x | 3.053 |
The honest caveat, which is also in the repo: this is a 32 KB prefix, not the full gigabyte, so these are not comparable to published full-enwik9 records. The prefix cuts against the classical coders rather than for them, since gzip and lzma earn their published numbers by building large dictionaries over long files and 32 KB gives them nowhere near enough room. Read the table as like-for-like at small scale, not as a record attempt.
The model size curve is the interesting part. Going from 82M to 494M parameters buys a 1.5x better ratio, and the ranking follows model quality, not model size: SmolLM2 at 135M beats gpt2-medium at 355M because it is simply the better predictor. Compression ratio turns out to be a clean, cheat-proof benchmark of how well a model actually understands text.
Two ways to spend the prediction, and why one loses
The obvious version of this idea, the one people usually reach for first, is to sort the vocabulary by probability and emit the index of the true token. Rank 0 dominates, the stream looks trivially compressible, and llmzip implements it properly with an adaptive arithmetic coder on top of the index stream rather than a naive entropy coder.
It still loses by around 10 percent. A rank throws away the model's confidence. "Most likely token" is not the same claim as "p = 0.9 rather than p = 0.3", and that difference is real bits that the rank representation has already discarded before the coder ever sees it. Coding against the full distribution keeps them.
Do not compress the output twice
Running a general-purpose compressor over an llmzip archive makes it bigger, every time. Measured on a 902-byte payload: zlib +1.2%, gzip +2.5%, lzma +6.9%, bzip2 +28%, brotli +0.4%, zstd +1.1%. Byte entropy of the payload is 7.767 out of 8, and genuinely random bytes of that length score 7.72 to 7.82, so there is no structure left to find.
That result is guaranteed rather than lucky, and it is the cleanest way to check the coder is working: if a second pass could shrink the output, the first pass was not optimal. The part that was worth shrinking was the header, a fixed cost on every archive and 11 percent of a 1 KB one. Packing the version metadata as a delimited string instead of JSON took it from 112 bytes to 69.
What it costs
- The model is the dictionary, and it is not in the archive. Both ends need the identical model. Archives are worthless without it.
- Decoding needs bit-exact reproducibility. A different dtype, thread count, device or torch version changes the probabilities in the last decimal place, and that is enough to desynchronise the arithmetic decoder and corrupt everything after it. The archive records all of them, pins the thread count on decode, and carries a digest. Compression round-trips its own output and refuses to write an archive that does not decode.
- It is slow. 6 to 45 tokens per second on CPU depending on the model, and decompression costs exactly as much as compression, since it runs the same forward passes. Ratio is the only axis where this wins, and it wins there by a lot.
- UTF-8 only. If the tokenizer is not byte-exact on the input, the archive falls back to raw LZMA rather than lose data.