# A transformer that never stops thinking

> A new architecture keeps one hidden state running across every token, so reasoning depth grows with the sequence instead of stopping at the last layer.
- Title: A transformer that never stops thinking · Github | Shortify
- Summary: A new architecture keeps one hidden state running across every token, so reasoning depth grows with the sequence instead of stopping at the last layer. Depth…
- Keywords: transformers, recurrence, latent reasoning, reinforcement learning, architecture, technology, startups, transformer, never, stops, thinking, Github
- Source: Github — https://yifanzhang-pro.github.io/recurrent-looped-tranformer
- Read time: 1 min
- Topics: transformers, recurrence, latent reasoning, reinforcement learning, architecture, technology, startups
## Encoder reads, decoder remembers
Recurrent Looped Transformer pairs a causal encoder that builds global key-value memory with a recurrent decoder that carries its final hidden state and sliding-window cache forward through every token.

> The encoder constructs global key, value memory; the decoder extends a continuous latent computation as the sequence grows.
## Depth that grows with length
Each new token pushes the recurrent path through the whole decoder again. After t tokens the path has crossed 48t decoder blocks, while per-token cost stays flat.

> Infinite depth refers to an extensible temporal path, not infinite work within a token.
## Parallel work around a serial core
Known tokens get encoded in one causal batch, weights and memory are reused, and activations are checkpointed. Only the decoder updates have to run strictly in order.
## No reset at the prompt boundary
Prompt and response share one state transition. The recurrent output and the decoder's sliding-window KV both survive the handoff from prefill to generation.

> Neither decoder state component resets at the serving boundary.
## One transition for training and RL
Pretraining, SFT, sampling, and replay all run the same update. RL replay rebuilds the whole history under current weights, since a parameter change invalidates every old cache.

> Behavior log-probabilities must describe the actual sampling distribution.
## Gradients have to flow everywhere
Full backprop through time touches recurrent outputs, decoder KV, and encoder memory. Detach any one of them and you are training a different model than you think.

> Forward consistency and complete gradients are separate requirements.
## Key takeaway

Depth does not have to live inside a single token. If the model carries its state forward, the reasoning path gets longer with every word it reads or writes.