IAAIIso-Depth Scaling Laws
How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Measures how much one recurrence is worth in equivalent unique parameters via a joint scaling law fitted on an iso-depth sweep over recurrence counts 1–8, yielding a recurrence-equivalence exponent φ = 0.46.
Inside the method
Prelude → [ Recurrent block ] × r → Coda, N_eff = N_once + r^φ · N_rec
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Recurrence r ∈ {1, 2, 4, 8}
- KV / state strategy
- See paper
Reading note
At r = 4 a 410M looped model matches a 580M non-looped model but costs as much to train as a 1B one. Truncated backprop lowers φ to 0.38 while hyper-connections raise it to 0.65, making φ a diagnostic for genuine loop improvements.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{isodepthscaling2026,
title = {How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models},
author = {Kristian Schwethelm and Daniel Rueckert and Georgios Kaissis},
year = {2026},
eprint = {2604.21106},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2604.21106}
}