IAAILooped Prediction Transformer
The Surprising Effectiveness of Shared Memory in Looped Transformers
Pretrains looped models in which only the first recursion writes a KV cache and later recursions read it while keeping a short local window, so memory does not grow with loops. Sharing memory improves rather than hurts quality.
Inside the method
Recursion 1 writes shared KV → recursions 2…R read it + short local window
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed recursions (five in the main results)
- KV / state strategy
- Shared first-recursion cache plus a short per-recursion window
Reading note
From 150M to 1B, the hybrid variant with five recursions lowers FineWeb-Edu perplexity by 1.12–1.82 versus a same-size Transformer while using 76–79% less context memory. Later recursions mostly attend to the shared memory, which also acts as a gradient highway.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopedpredictiontransformer2026,
title = {The Surprising Effectiveness of Shared Memory in Looped Transformers},
author = {Giovanni Monea and Keshav Ramji and Yousef El-Kurdi and Luis A. Lastras and Yoav Artzi and Nathan Godey and Ramón Fernandez Astudillo},
year = {2026},
eprint = {2610.02383},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02383}
}