← Back to the collection
Full-stack recurrence ·
Residual Scaling
On the Residual Scaling of Looped Transformers: Stability and Transferability
Analyzes how weight sharing correlates residual updates across loop iterations and derives a 1/N residual scale for a block repeated N times. Factoring loop count from unique-layer depth improves training stability and supports learning-rate transfer across loop counts.
Inside the method
[ Residual-scaled shared block ] × N
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Configurable N loops over L unique layers
- KV / state strategy
- See paper
Reading note
For L unique layers repeated N times, the parameterization is epsilon = lambda / (N sqrt(L)). The analysis separates correlation across loops from variance across unique layers; Transformer experiments compare loss and learning-rate transfer across loop counts at fixed unique-layer depth.
Sources checked 2026-09-15. This catalog does not imply independent reproduction.
Cite this work
@misc{residualscaling2026,
title = {On the Residual Scaling of Looped Transformers: Stability and Transferability},
author = {Shaowen Wang and Bingrui Li and Ge Zhang and Wenhao Huang and Shen Yan and Jian Li},
year = {2026},
eprint = {2606.18524},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.18524}
}