IAAILSTM-Style Looped Transformer
LSTM-Style Looped Transformers with Shared KV Caches
Studies an LSTM-style gated recurrent transformer whose recurrent core also shares a single KV cache. A consistency loss late in training aligns the cache entries seen in parallel training with those produced recursively at generation time.
Inside the method
[ LSTM-style gated core ] × R (one shared KV cache)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed recurrence
- KV / state strategy
- One KV cache shared across the recurrent core; persistent cache sets drop from 15 to 4
Reading note
At width 1024 it uses 40.4% fewer parameters than an untied control at nearly matched FLOPs. The consistency loss removes 81–89% of the continuation-loss penalty on FineWeb; the cost of weight tying is smallest at low token budgets.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{lstmsharedkv2026,
title = {LSTM-Style Looped Transformers with Shared KV Caches},
author = {Andreas Kirsch},
year = {2026},
eprint = {2610.33360},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33360}
}