IAAIDo Halting Heads Save Compute?
Do Halting Heads Really Save Compute in Looped Language Models?
Pretrains looped models up to 1.3B parameters to test learned halting heads. Heads trained on a mix of iterations predict early stops yet tokens run almost every iteration under the usual rule, and the commonly reported average-iterations metric overstates savings in batched generation.
Inside the method
[ Shared layers ] → halting head after each iteration
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- Learned per-token halting (evaluated critically)
- KV / state strategy
- See paper
Reading note
Scoring every iteration fixes the head only with a strong early-stop penalty. The authors recommend reporting iterations actually executed with threshold and batch size, and provide a pre-training test that bounds a head's possible gain.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{haltingheads2026,
title = {Do Halting Heads Really Save Compute in Looped Language Models?},
author = {Tsung-Han Wu and Minsuk Heo and Kenny Tan},
year = {2026},
eprint = {2610.32246},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.32246}
}