A Transformer that loops itself into an RNN — and generalizes

What it does Recurrent Looped Transformer (RLT) is a research architecture from Princeton and UPenn that runs the same Transformer decoder once per token, feeding the previous token’s final decoder state into the current token’s input through a gated merge. A parallel causal encoder supplies token representations and a global KV memory, while the decoder keeps a sliding-window attention cache at every layer. The result: after t tokens the recurrent path has traversed t·L_D decoder blocks, yet the per-token cost stays fixed.

The interesting bit RLT-1 is literally an RNN whose cell is “gated merge + full Transformer decoder” — but unlike S4 or Mamba, the state transition is nonlinear in the previous state, so the usual parallel-scan tricks don’t apply. The authors also explore the feedback interval: RLT-2 updates the state only once per chunk of B tokens (trainable in parallel, reducible to B=1 later), and RLT-0 removes feedback entirely — 787,968 fewer parameters at the eight-layer split.

Key highlights

• Parity: trained on ≤40 bits, 5+3 and 7+1 splits hit 100 ± 0% at 256 bits; a standard 8-layer Transformer gets 50.07 ± 1.63%.
• Swaps-S5 at 256 operations (8× training length): 4+4 split reaches 97.30 ± 2.76% vs 0.85% for the Transformer.
• Feedback frequency matters a lot: chunk4 loses ~1 point on 64-bit parity but ~80 points on 64-operation swaps-S5; RLT-0 sits near chance on parity.
• Chunk size can vary during training — large chunks for parallel pretraining, shrinking toward B=1 — though the experiments train each size from scratch.
• All comparisons use three seeds with shared data and test sets, reported as mean ± sample SD.

Caveats

• These are small algorithmic tasks (parity, addition, mod-5, S5 permutations) with ~26–29M-parameter models; no language modeling or real-world benchmarks appear in the README.
• Standard S5 stays near the uniform reference for every model, and flat mod-5 results vary wildly across seeds (e.g. 44.43 ± 43.81%).
• Equal layer counts don’t match parameters or compute against the GPT baseline, as the authors themselves note.

Verdict Worth a look if you care about length generalization, recurrent hybrids, or the Mamba/looped-Transformer design space. Skip it if you want a drop-in LLM replacement — this is a carefully instrumented research probe, not a product.