# Can gzip be a language model?

> Gzip generates coherent text without neural networks by predicting byte sequences.
- Title: Can gzip be a language model? · Nathan | Shortify
- Summary: Gzip generates coherent text without neural networks by predicting byte sequences. Gzip can create text by compressing prompts using its internal probability…
- Keywords: gzip, language model, compression, deflate, beam search, shakespeare, technology, startups, language, model, Nathan
- Source: Nathan — https://nathan.rs/posts/gzip-lm
- Read time: 1 min
- Topics: gzip, language model, compression, deflate, beam search, shakespeare, technology, startups
## Language modeling without neural networks
The author previously generated Shakespeare using an unbounded n-gram model with no weights or training.
## Compression-prediction equivalence
Every prediction model is inherently a compressor, and all compression algorithms are prediction models.

> every prediction model is inherently a compressor, and all compression algorithms are prediction models
## Gzip as a language model
Gzip can generate text by priming it with a corpus and continuing prompts using byte sequences that compress best.
## Real output from gzip
Gzip generated Shakespeare-like text after priming on tiny Shakespeare: 'MENENIUS: 'Thou canq' and 'Pray now, nocamest thou to a morsel.'

> MENENIUS:
'Though all at once canq
## Compression is prediction
Compressors predict data by spending few bytes on expected patterns and many bytes on unexpected data.

> It spends few bytes on data it expects and many bytes on data it doesn't.
## Generating by beam search
Gzip uses beam search to find the most compressible byte sequences, avoiding quantization noise from single-byte predictions.

> The fix is to look ahead a whole span before committing. gzipt runs a beam search over byte sequences.
## What happens next
The code is available on GitHub for playing with, and the method improves text generation significantly.
## Key takeaway

Gzip can create text by compressing prompts using its internal probability model.