From Nathan · · 1m 10s
Can gzip be a language model?
Gzip generates coherent text without neural networks by predicting byte sequences.
In brief
Gzip generates coherent text without neural networks by predicting byte sequences. Gzip can create text by compressing prompts using its internal probability model. Originally reported by Nathan.
Language modeling without neural networks
The author previously generated Shakespeare using an unbounded n-gram model with no weights or training.
Compression-prediction equivalence
Every prediction model is inherently a compressor, and all compression algorithms are prediction models.
“every prediction model is inherently a compressor, and all compression algorithms are prediction models”
Gzip as a language model
Gzip can generate text by priming it with a corpus and continuing prompts using byte sequences that compress best.
Real output from gzip
Gzip generated Shakespeare-like text after priming on tiny Shakespeare: 'MENENIUS: 'Thou canq' and 'Pray now, nocamest thou to a morsel.'
“MENENIUS: 'Though all at once canq”
Compression is prediction
Compressors predict data by spending few bytes on expected patterns and many bytes on unexpected data.
“It spends few bytes on data it expects and many bytes on data it doesn't.”
Generating by beam search
Gzip uses beam search to find the most compressible byte sequences, avoiding quantization noise from single-byte predictions.
“The fix is to look ahead a whole span before committing. gzipt runs a beam search over byte sequences.”
What happens next
The code is available on GitHub for playing with, and the method improves text generation significantly.
The one thing
Gzip can create text by compressing prompts using its internal probability model.
