← Back to the collection
Core recurrence ·
SMELT
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Repeats the middle half of a sparse MoE Transformer's layers twice while closely matching per-token FLOPs, total non-embedding parameters and KV cache to an unlooped baseline. Budget-matched scaling experiments show improved loss and downstream performance from reusing the shared middle layers.
Inside the method
Front layers → [ Middle half ] × 2 → Back layers
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed: middle half of layers repeated twice
- KV / state strategy
- Matched to the unlooped baseline
Reading note
Separate fitted Chinchilla-style scaling laws estimate 6.8–18.0% training-FLOP savings on the compute-optimal frontier. Experiments cover four sizes up to 54B non-embedding parameters, with the largest domain gains on Code and stronger gains for longer samples and more in-context examples.
Sources checked 2026-09-15. This catalog does not imply independent reproduction.
Cite this work
@misc{smelt2026,
title = {SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers},
author = {Shaowen Wang and Ge Zhang and Kairong Luo and Yuhao Wu and Shaofan Liu and Jiaheng Liu and Wenhao Huang and Shen Yan and Jian Li},
year = {2026},
eprint = {2609.01343},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.01343}
}