IAAIGated Recurrent Transformer
Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
A prelude–core–coda recurrent-depth transformer whose shared core update is modulated by a lightweight projection and an elementwise update gate conditioned on the hidden state, the prelude output and per-step noise, letting a few layers specialize across recurrences.
Inside the method
Prelude → [ Core + GRU-style update gate ] × R → Coda
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Recurrence count R with resampled noise
- KV / state strategy
- See paper
Reading note
At iso-FLOPs, a 3-layer model matches a 12-layer GPT-2 Small and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells. At iso-parameters, deeper recurrence reaches 2.76 validation loss versus 2.84.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{gatedrecurrenttransformer2026,
title = {Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation},
author = {Amr Hegazy and Amr Alanwar and Mostafa Elhoushi},
year = {2026},
eprint = {2608.15062},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.15062}
}