IAAILSTM-UT
Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers
Shows on cellular-automaton tasks that keeping more history is not the same as making it usable: CoTFormer's full cache can pull corrected trajectories back toward failure. LSTM-UT adds a small, bounded, gated cell state to the shared block instead.
Inside the method
[ Shared block + gated cell state ] × N
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Depth extrapolation far beyond training
- KV / state strategy
- Small bounded cell state rather than a cache of every step
Reading note
Trained to depth 12 on Rule 30, it keeps 99.7% exact-row accuracy at depth 60 where a Block Universal Transformer gets no row right, and one checkpoint stays above 99.95% at depth 1,000. On OpenWebText2 it beats BUT and roughly matches CoTFormer.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{lstmut2026,
title = {Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers},
author = {Aras Kavuncu and Muhammad Burhan Hafez},
year = {2026},
eprint = {2609.19521},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.19521}
}