IAAILOOM
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Diagnoses why looped MoE models stop improving after two loops: variance growth across iterations and expert-selection collapse. LOOM scales residual updates, re-injects the input embedding every loop, gives each loop its own router, and adds a looping residual that carries earlier outputs forward.
Inside the method
[ MoE block, per-loop router ] × R + input re-injection + looping residual
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Stable scaling to 9–12 loops
- KV / state strategy
- See paper
Reading note
Across 100M–1.7B models; under near-iso-FLOP the 700M model is best at 5 loops (perplexity 18.36 → 16.54). Without FLOP matching, the 1.7B model trained on 60B tokens peaks at 9 loops (perplexity 9.62 → 7.77).
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loom2026,
title = {Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts},
author = {Di He and Pengxiang Li and Da Chang and Qingyan Meng and Lu Yin and Shiwei Liu},
year = {2026},
eprint = {2610.01153},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01153}
}