IAAILatent Recurrent Transformer
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
Reuses a high-level hidden state from the previous token as recurrent memory for the next token, adding a cross-token, cross-layer pathway while keeping standard attention, KV cache and one forward per token. Interleaved parallel training avoids sequential unrolling during pretraining.
Inside the method
token t−1: high-level state → input of token t (one forward per token)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Recurrence across tokens
- KV / state strategy
- Standard KV cache interface
Reading note
On 1.3B and 2.1B nanochat-style backbones it improves BPB and CORE at matched effective compute, beats two-forward PonderLM-2 and matches a three-loop Transformer in BPB with 9% latency overhead.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{latentrecurrenttransformer2026,
title = {Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior},
author = {Zeyi Huang and Xuehai He and LiLiang Ren and Yiping Wang and Baolin Peng and Hao Cheng and Shuohang Wang and Pengcheng He and Jianfeng Gao and Yong Jae Lee and Yelong Shen},
year = {2026},
eprint = {2605.26797},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.26797}
}