IAAICompute ≠ Retain ≠ Reuse
Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers
Separates three causes of overthinking in looped transformers on in-context Mealy-machine execution: failed computation, lost retention and non-reusable states. The computation itself is exact, but answers decay with extra loops unless training asks for retention.
Inside the method
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Trained to answer at loop 32; probed up to loop 64
- KV / state strategy
- See paper
Reading note
Models keep only 22% of answers at loop 64 by default; supervising a random position keeps 99.8% and randomized loop counts help, while gradient truncation does not. Even retained states are not reusable when the answer first appears. Synthetic tasks.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{computeretainreuse2026,
title = {Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers},
author = {Peilong Zhou and Hongtao Zhang and Yulong Huang and Cangyuan Li and Mingyu Chen and Ying Wang},
year = {2026},
eprint = {2610.19768},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.19768}
}