IAAISHIFT
What Makes Looped Transformers Perform Better Than Non-Recursive Ones
Explains why looped transformers beat non-recursive ones on complex reasoning through loss-landscape geometry: recursion biases training toward steep V-shaped river valleys, where valley hopping speeds convergence along the river. This motivates SHIFT, a staged progressive training schedule.
Inside the method
[ Shared attention block ] × L — staged hierarchical training
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed loops
- KV / state strategy
- See paper
Reading note
The landscape account is a conjecture supported by sample- and Hessian-level observations. SHIFT accelerates training of looped models while reaching comparable performance.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{shift2025,
title = {What Makes Looped Transformers Perform Better Than Non-Recursive Ones},
author = {Zixuan Gong and Yong Liu and Jiaye Teng},
year = {2025},
eprint = {2510.10089},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2510.10089}
}