IAAILooped Transformers for Length Generalization
Looped Transformers for Length Generalization
Shows that looped transformers with an adaptive number of steps generalize to unseen input lengths on tasks with known iterative solutions built from repeated RASP-L operations, using a dedicated training algorithm.
Inside the method
[ Looped block ] × n(input length)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- Steps scale with input length
- KV / state strategy
- See paper
Reading note
Learns highly length-generalizable solutions on tasks such as parity and addition where standard transformers fail to extrapolate.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopedlengthgeneralization2024,
title = {Looped Transformers for Length Generalization},
author = {Ying Fan and Yilun Du and Kannan Ramchandran and Kangwook Lee},
year = {2024},
eprint = {2409.15647},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2409.15647}
}