IAAIOperLoop
Looped Transformers as Optimizers
Views the looped hidden state as a fast weight and derives loop transitions in closed form from a projection, a local objective and an optimizer update rule. This exposes mismatches in existing transitions and yields OperLoop, with explicit weight decay, adaptive step size and a delta objective.
Inside the method
[ Block ] × R where each step = local gradient update on an implicit target
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed loops
- KV / state strategy
- See paper
Reading note
Aligning input maps of existing transitions lowers training loss and improves commonsense accuracy; OperLoop improves average generative performance over looped and non-looped baselines at matched training FLOPs.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{operloop2026,
title = {Looped Transformers as Optimizers},
author = {Yulong Huang and Chen Jiang and Zhanpeng Zhou and Hongtao Zhang and Tianyu Li and Tianyu He and Xiangyu Zhang and Bojun Cheng},
year = {2026},
eprint = {2609.37379},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.37379}
}