IAAILoop Dropout
Loop Dropout: Regularizing Shared Updates in Looped Language Models
Finds that standard LoRA on looped models mostly adapts late loops. Loop Dropout randomly masks adapter applications per loop during training and rescales by inverse survival probability, so the shared update stays strong and effective at every loop.
Inside the method
[ Block + LoRA (randomly masked, rescaled) ] × R
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- All loops active at inference
- KV / state strategy
- See paper
Reading note
Improves mathematical reasoning across sizes, ranks and recipes, extending to instruction tuning and code. Inference uses standard LoRA at all loops with no extra parameters or compute.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopdropout2026,
title = {Loop Dropout: Regularizing Shared Updates in Looped Language Models},
author = {Zirui Zhu and Hailun Xu and Xuanlei Zhao and Yong Liu and Yingxuan Ren and Kanchan Sarkar and Kun Xu and Yang You},
year = {2026},
eprint = {2609.34218},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.34218}
}