IAAILoopMoE
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
A looped mixture-of-experts language model with IterAdaLN, a modulation conditioned on iteration index and per-token state to break weight-sharing symmetry, and a capacity-balancing strategy that restores the attention-to-FFN active-parameter ratio of tuned non-looped models.
Inside the method
[ MoE block + IterAdaLN(iteration, state) ] × R
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed iterations
- KV / state strategy
- See paper
Reading note
Strictly controlled against a vanilla MoE with identical total parameters, per-token FLOPs and active sublayer ratios, its nine-benchmark advantage grows from over 1 point at 3B to about 3 points at 9B.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopmoe2026,
title = {LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling},
author = {Wenkai Chen and Tianshu Li and Wenyong Huang and Yichun Yin and Lifeng Shang and Chengwei Qin},
year = {2026},
eprint = {2606.04438},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.04438}
}