IAAITruncate or Not?
Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
A scaling study of truncated backpropagation through time in prelude–recurrent–coda looped transformers. Nearly 600 runs jointly vary model size, tokens and forward/backward loop counts, and a unified law expresses looping and truncation through effective parameters.
Inside the method
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed forward loops T with B backward loops
- KV / state strategy
- See paper
Reading note
When forward and backward loop counts are optimized jointly, the law favors full BPTT; in matched-compute runs, full BPTT through fewer loops beats truncation. Two fully trained loops beat the compute-optimal non-looped model with about 27% fewer non-embedding parameters and 8% more training compute.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{truncateornot2026,
title = {Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation},
author = {Xuyuan Xiong and Siyu Chen and Zihan Shao and Yue Su and Weiyang Jin and Ping Luo and Zhuoran Yang and Xihui Liu},
year = {2026},
eprint = {2610.26030},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.26030}
}