IAAIBeyond the Training Horizon
Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers
A mechanistic comparison of two recurrence-training schemes for looped transformers on polynomial iteration, finite-state composition and knowledge-graph traversal, revealing distinct mechanisms and why both fail at greater depths.
Inside the method
[ Block ] × R — matched vs decoupled recurrence training
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Length generalization beyond trained depth
- KV / state strategy
- See paper
Reading note
Recurrent states encode both content and its computational status, and transferable residual directions causally control whether information can be used in later steps. Length generalization need not rely on faithful step-by-step reasoning.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopedlengthgeneralizationmechanisms2026,
title = {Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers},
author = {Jia Liang and Xi Jin and Liangming Pan},
year = {2026},
eprint = {2609.33144},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.33144}
}