IAAIFoil
How to Loop MoE: Flatten the Experts, Untie the Attention
Answers how to loop a mixture-of-experts model at fixed expert parameters and compute: flatten the experts by halving expert layers, doubling experts per layer and doubling passes, and untie attention so each pass has its own attention while experts and routers stay shared.
Inside the method
[ Untied attention + shared wide MoE ] × 2k (flattened from k layers)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- More passes over fewer, wider expert layers
- KV / state strategy
- Per-pass attention parameters
Reading note
At 100B tokens, loss improves monotonically with flattening; the most flattened model ends 0.012 nat below the looped baseline at equal parameters and compute. Untying attention gives more balanced, confident routing.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{foil2026,
title = {How to Loop MoE: Flatten the Experts, Untie the Attention},
author = {Shouren Wang and Chuang Ma and Mohsen Hariri and Debargha Ganguly and Wang Yang and Xiaoqing Tong and Qianying Liu and Xiaotian Han and Vipin Chaudhary},
year = {2026},
eprint = {2609.35751},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.35751}
}