IAAIUniversal Transformers Need Memory
Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
Studies learned memory tokens as a scratchpad for a single-block Universal Transformer with ACT on Sudoku-Extreme, finding them necessary and showing that memory size and ponder depth substitute for each other at fixed accuracy.
Inside the method
[ Single block + memory tokens ] × r (ACT halting)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- Adaptive Computation Time
- KV / state strategy
- Learned memory tokens
Reading note
No configuration without memory tokens works; 8–32 tokens reach 57.4% exact match. Identifies an ACT router initialization trap and fixes it with a negative bias ('deep start'); ACT then has far lower seed variance than fixed depth.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{utneedmemory2026,
title = {Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning},
author = {Grigory Sapunov},
year = {2026},
eprint = {2604.21999},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2604.21999}
}