IAAIRetrofitting Recurrent Depth
Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
Retrofits Qwen2.5-0.5B-Instruct into prelude, weight-tied recurrent block and coda with an identity-preserving one-loop path and a re-entry bridge, and shows it learns a reusable iterative latent procedure, either as a 6M adapter over frozen weights or a 180M full block.
Inside the method
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- One task step per loop; extrapolates to about 1.5× supervised depth
- KV / state strategy
- See paper
Reading note
Non-inferior to the base at one loop on a preregistered ARC battery. Against a same-size scratchpad model it wins overall (84% versus 72%), retains far more beyond depth 10, and answers 7.6× faster. Single-author study on synthetic tasks.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{retrofitrecurrentqwen2026,
title = {Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets},
author = {Mark Shapiro},
year = {2026},
eprint = {2608.11233},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.11233}
}