IAAITrading Depth for Time
Trading Depth for Time in Recurrent Transformers
Compares spending extra compute on temporal recurrence versus physical depth. Latent Recurrent Transformers feed each token's high-level hidden state into the next, and inserting one latent thought token between vocabulary tokens adds a refinement pass through the shared backbone.
Inside the method
token t → [ Backbone ] → thought step → [ Backbone ] → token t+1 (state fed forward)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- One extra latent step per vocabulary token
- KV / state strategy
- See paper
Reading note
On NanoChat MoE backbones at matched training FLOPs, it beats PonderLM-2 and overtakes a two-loop Transformer as compute grows. One thought token recovers about 89% of the gain from doubling depth with about 48% fewer parameters.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{latentrecurrenttransformertime2026,
title = {Trading Depth for Time in Recurrent Transformers},
author = {Zeyi Huang and Xuehai He and Yelong Shen and Yong Jae Lee},
year = {2026},
eprint = {2609.21605},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.21605}
}