IAAIUpLoop
UpLoop: Upcycling Large Language Models with Gated Loop
A post-training recipe that upcycles a pretrained LLM into a looped one by looping selected layers locally rather than the whole stack, with hidden-wise gates on attention and MLP residuals so later loops refine instead of rewrite.
Inside the method
Selected layer: [ gated attention + gated MLP ] × R
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- R = 3 main setting; native and extrapolated R = 4
- KV / state strategy
- See paper
Reading note
Starting from Qwen3-4B Base with SFT, R = 3 improves the 20-benchmark average by 6.17%. The authors report breaking the common saturation at two loops.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{uploop2026,
title = {UpLoop: Upcycling Large Language Models with Gated Loop},
author = {Tieyuan Chen and Huabin Liu and Yingtao Zhang and Shijie Li and Tengyu Pan and Weiyao Lin and Jianguo Li},
year = {2026},
eprint = {2610.25004},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.25004}
}