IAAILoopFold
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Cuts the sequential latency of deep looped models by folding several recurrent depths into one parallel batch for the same prefix. A lightweight predictor guesses the inputs of later loops so they can start early, and a corrector refines them once earlier loops finish; everything is trained with the ordinary next-token loss.
Inside the method
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed loop count executed in fewer sequential stages
- KV / state strategy
- See paper
Reading note
With 1.4B models, deeper LoopFold settings raise the eight-task zero-shot average from 53.41% to 56.38%, where vanilla looping and PLT gain little. The 46% decoding-latency reduction versus a nine-loop model uses five sequential stages plus extra parallel compute.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopfold2026,
title = {LoopFold: Parallelizing Recurrent Depth in Looped Transformers},
author = {Feichen Song and Boyi Zeng and Yiqin Hao and Shixiang Song and He Li and Yizhou Yang and Zitong Wang and Yifan Liu and Ziwei He and Zhouhan Lin},
year = {2026},
eprint = {2610.35556},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.35556}
}