IAAIRecurrent Looped Transformer
Recurrent Looped Transformer
Splits the layers into a parallel causal encoder and a recurrent decoder that merges each token's encoder output with the previous token's final decoder state, so the computation path grows with sequence length at a fixed per-token cost.
Inside the method
Causal encoder (parallel) → recurrent decoder fed by previous token's state
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Grows with sequence length; fixed per token
- KV / state strategy
- See paper
Reading note
Trained on at most 40 bits, two splits generalize parity to 256 bits in every seed; on swap-S5 at 8× training length it reaches 97% versus under 1% for a Transformer. Removing the feedback drops these tasks to chance.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{rlt2026,
title = {Recurrent Looped Transformer},
author = {Yifan Zhang and Jichen Feng and Shihan Qin and Sun-Yuan Kung and Mengdi Wang},
year = {2026},
eprint = {2610.07591},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07591}
}