IAAIT²MLR
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
Temporal Middle-Layer Recurrence fuses a cached middle-layer representation from the previous token into an earlier layer at the current position, letting intermediate computation persist across decoding steps with little overhead and without looping all layers.
Inside the method
token t−1: middle layer → fused into early layer of token t
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- One forward per token; recurrence across time
- KV / state strategy
- Cached middle-layer state from the previous token
Reading note
Beats data- and parameter-matched Transformers in pretraining and multi-hop reasoning; recurrence over as little as 20% of the network often beats full-layer recurrence. Can be retrofitted into a pretrained 1.7B model with brief fine-tuning.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{t2mlr2026,
title = {T^2MLR: Transformer with Temporal Middle-Layer Recurrence},
author = {Ziyang Cai and Xingyu Zhu and Yihe Dong and Yinghui He and Sanjeev Arora},
year = {2026},
eprint = {2607.15178},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.15178}
}