IAAISparse Layers for Looped Scaling
Sparse Layers are Critical to Scaling Looped Language Models
Compares dense and MoE transformers with and without looping and finds that looped MoE models scale better than the standard baseline while dense looped models do not, because different experts fire on each pass through the shared layers.
Inside the method
[ Shared MoE layers ] × R (routing diverges across loops) + exits at loop boundaries
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Early exits at loop boundaries
- KV / state strategy
- See paper
Reading note
Loop boundaries make better exit points than intermediate layers of standard models, with earlier output convergence. Recommends looped MoE with early exits for memory and inference savings.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{sparselayersloopedscaling2026,
title = {Sparse Layers are Critical to Scaling Looped Language Models},
author = {Ryan Lee and Jacob Biloki and Edward J. Hu and Jonathan May},
year = {2026},
eprint = {2605.09165},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.09165}
}