From GitHub · · 3m
mini-AGI: Continual Learning Model on 8GB VRAM Laptop
A continual learning model trained from scratch on 8GB VRAM laptop with batch-1 stream of data.
In brief
A continual learning model trained from scratch on 8GB VRAM laptop with batch-1 stream of data. mini-AGI is a byte-level language model that trains from scratch on a single 8GB VRAM GPU and keeps learning from everything it reads without catastrophic forgetting. Originally reported by GitHub.
What is mini-AGI
mini-AGI is a continual learning byte-level language model that assembles its own architecture, trains from scratch on a single 8 GB VRAM GPU, and keeps learning from everything it reads.
How it stores weights
It stores its weights as ordinary files on disk and pages them onto the card as it needs them, so the parameter count is bounded by free disk space rather than by VRAM.
Target hardware
Targeted at a PC or laptop with at least an 8 GB VRAM GPU on the board.
Current capabilities
As of now this is a small toy-level model. Do not expect frontier level capabilities.
Why it matters
This is rather a small experiment to show that continual learning from a single stream of data without catastrophic forgetting is possible on modest hardware.
How it works
The model reads a stream of characters one chunk at a time, takes a gradient step on each, and the same path serves generation.
Three constraints
It has to fit on 8 GB. It has to not forget. It has to be able to read anything.
Architecture
Characters pass through two dense prelude blocks and then through one recurrent block applied up to 24 times, each application choosing its own experts from a shared pool.
Adaptive depth
A halting head scores every character at every row, and the character stops as soon as another row would not change the answer.
Routing per block
Each recurrent application a character runs picks its own top-8 experts from the 32 on the card, so one character touches far more of the pool than top-8 suggests.
No expert labels
There are no labels anywhere. Soft top-k routing distributes capability across the pool by itself, and a character can combine fragments from several experts.
Paging works
Every expert is a file on disk holding its weights and its Adam moments. Above disk sit a RAM cache and the card.
How growth works
New experts are added on speculation, at a small gate so they change almost nothing, and kept only if something goes on asking for them.
How pruning works
A new expert is safe for a full survival window no matter what, so it cannot be judged before it has had a chance to be chosen.
Continual learning test
The model was switched cold onto PG19 - 19th-century novels, a domain it has never read - and made to read 1,048,576 consecutive characters of it and nothing else.
Results
The mitigated configuration reached 97.30% retention on the eight subjects it already knew while adding a new domain without forgetting anything.
The one thing
mini-AGI is a byte-level language model that trains from scratch on a single 8GB VRAM GPU and keeps learning from everything it reads without catastrophic forgetting.
