IAAIScaleRLT
ScaleRLT: Exact Log-Probabilities and Fast Deferred Rollouts for Reinforcement Learning on Looped Transformers
Makes on-policy RL practical for looped language models such as Ouro. Rollouts defer and batch the deeper loops, and the learner uses a hybrid log-probability whose value replays the rollout schedule exactly while its gradient comes from a dense pass, removing the floating-point mismatch between sampler and learner.
Inside the method
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Ouro-style fixed loops; mixed-depth or native readout
- KV / state strategy
- One-wave KV admission with CUDA graphs
Reading note
Replay is bit-exact in all reported FP32 and bf16 cells. Deferral raises single-GPU rollout throughput 1.47–1.56× and a two-node GRPO update is 2.71× faster. The GSM8K gains come from 100-update runs in the authors' setup.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{scalerlt2026,
title = {ScaleRLT: Exact Log-Probabilities and Fast Deferred Rollouts for Reinforcement Learning on Looped Transformers},
author = {Dezhen Lu and Zixiang Ni and Runguo Li},
year = {2026},
eprint = {2610.34427},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.34427}
}