IAAILost Gradient under Truncation
How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
Measures exactly how much weight gradient truncated backprop discards for every truncation length, across nine looped language models including Parcae and Ouro.
Inside the method
Backprop through the last k of R loops → measure discarded gradient
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Analysis of truncated BPTT
- KV / state strategy
- See paper
Reading note
The lost share is set near the output and barely changes once R is well above k; on Ouro, keeping two of four loops still loses 81% of the output gradient. The share falls during fine-tuning, and the paper could not show it predicts training cost better than simple rules.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{truncatedbackpropgradient2026,
title = {How Much Gradient Does Truncated Backprop Lose in Looped Transformers?},
author = {Aur Shalev Merin},
year = {2026},
eprint = {2610.28871},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28871}
}