IAAIRetrofitted Recurrence
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
Converts pretrained non-recurrent language models such as TinyLlama, OLMo-2 and Llama-3.2 into depth-recurrent prelude–core–coda models, using a curriculum that gradually increases recurrences during training.
Inside the method
Pretrained layers → Prelude → [ Recurrent core ] × r (curriculum) → Coda
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Recurrence increased by curriculum; variable at test time
- KV / state strategy
- See paper
Reading note
On mathematics, converted recurrent models beat simply post-training the original model at the same compute budget. Released checkpoints include Recurrent-OLMo and Recurrent-TinyLlama.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{retrofittedrecurrence2025,
title = {Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence},
author = {Sean McLeish and Ang Li and John Kirchenbauer and Dayal Singh Kalra and Brian R. Bartoldson and Bhavya Kailkhura and Avi Schwarzschild and Jonas Geiping and Tom Goldstein and Micah Goldblum},
year = {2025},
eprint = {2511.07384},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2511.07384}
}