From Github · · 1m 30s
A transformer that never stops thinking
A new architecture keeps one hidden state running across every token, so reasoning depth grows with the sequence instead of stopping at the last layer.
In brief
A new architecture keeps one hidden state running across every token, so reasoning depth grows with the sequence instead of stopping at the last layer. Depth does not have to live inside a single token. If the model carries its state forward, the reasoning path gets longer with every word it reads or writes…
Encoder reads, decoder remembers
Recurrent Looped Transformer pairs a causal encoder that builds global key-value memory with a recurrent decoder that carries its final hidden state and sliding-window cache forward through every token.
“The encoder constructs global key, value memory; the decoder extends a continuous latent computation as the sequence grows.”
Depth that grows with length
Each new token pushes the recurrent path through the whole decoder again. After t tokens the path has crossed 48t decoder blocks, while per-token cost stays flat.
“Infinite depth refers to an extensible temporal path, not infinite work within a token.”
Parallel work around a serial core
Known tokens get encoded in one causal batch, weights and memory are reused, and activations are checkpointed. Only the decoder updates have to run strictly in order.
No reset at the prompt boundary
Prompt and response share one state transition. The recurrent output and the decoder's sliding-window KV both survive the handoff from prefill to generation.
“Neither decoder state component resets at the serving boundary.”
One transition for training and RL
Pretraining, SFT, sampling, and replay all run the same update. RL replay rebuilds the whole history under current weights, since a parameter change invalidates every old cache.
“Behavior log-probabilities must describe the actual sampling distribution.”
Gradients have to flow everywhere
Full backprop through time touches recurrent outputs, decoder KV, and encoder memory. Detach any one of them and you are training a different model than you think.
“Forward consistency and complete gradients are separate requirements.”
The big picture
Depth does not have to live inside a single token. If the model carries its state forward, the reasoning path gets longer with every word it reads or writes.
