IAAILatentLoop
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Finds that in a looped MoE that applies a shared block twice, the first visit can be scaled down without harm while the second cannot. LatentLoop compresses the sequence on the first visit, keeps full resolution on the second, and spends the savings on a larger hidden size.
Inside the method
Visit 1: [ Block ] on compressed sequence → Visit 2: [ Block ] at full resolution
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Two visits with asymmetric compute
- KV / state strategy
- Separate attention per visit; MoE experts shared across visits
Reading note
Across 15 configurations from 1.3B to 7B parameters and beyond 10^21 FLOPs it lowers validation NLL; at a fitted NLL of 1.988 the predicted compute savings are 19.0% versus vanilla MoE and 7.9% versus the looped baseline.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{latentloop2026,
title = {LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation},
author = {Zefeng He and Yunkai Chai and Xiaoyang Chen and Tong Zhu and Zongliang Wu and Yun Luo and Yafu Li and Yu Cheng},
year = {2026},
eprint = {2610.19975},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.19975}
}