IAAITraining-Free Looped Transformers
Training-Free Looped Transformers
Loops a contiguous mid-stack block of a frozen checkpoint at inference with no training. Viewing a pre-norm block as a forward Euler step, it replaces one large update with smaller damped sub-steps, since naive block repetition usually hurts.
Inside the method
Lower layers → [ Mid block, damped sub-steps ] × k → upper layers (frozen weights)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Inference-time wrapper
- KV / state strategy
- See paper
Reading note
Across seven dense, MoE and MLA+MoE families, it improves Qwen3-4B-Instruct by 2.64 points on MMLU-Pro, Qwen3-30B-A3B-Instruct by 1.14 on CommonsenseQA and Moonlight-16B-A3B by 1.20 on OpenBookQA.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{trainingfreelooped2026,
title = {Training-Free Looped Transformers},
author = {Lizhang Chen and Jonathan Li and Chen Liang and Ni Lao and Qiang Liu},
year = {2026},
eprint = {2605.23872},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.23872}
}