IAAILearning to Extrapolate in Depth
Learning to Extrapolate in Depth with Looped Transformers
A theoretical and empirical study of how far a looped block trained at depth K can be run deeper, using all-pairs graph connectivity. Training converges to an update governed by one effective parameter, which yields a finite extrapolation window with explicit endpoints.
Inside the method
[ Block ] × K at training → × K′ at test (within a certified window)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Test depth chosen from the training distribution
- KV / state strategy
- See paper
Reading note
Too few test loops under-propagate and too many create false connections. The certified depth gain grows exponentially as training depth outpaces training hardness. Graph-connectivity setting.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{depthextrapolationwindow2026,
title = {Learning to Extrapolate in Depth with Looped Transformers},
author = {Huitao Yang and Jianhao Huang and Zixuan Wang and Baharan Mirzasoleiman},
year = {2026},
eprint = {2610.14456},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.14456}
}