IAAIAdaptive Depth as a Finishing Stage
Adaptive-Depth Is a Finishing Stage
Introduces fixed-first training: train a looped model at one fixed depth for most of the run and vary depth only in a short final phase, instead of training across depths from the start.
Inside the method
Fixed depth (most of training) → mixed depths (finishing phase)
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- Adaptive inference depth learned late in training
- KV / state strategy
- See paper
Reading note
At 200M, fixed depth 2 for the first 67% cuts modeled training compute by 45% with mean NLL across depths within 0.5%. At 1B, starting at depth 4 cuts compute by 31–37% and wall-clock time by 32–39% with final NLL within 0.8%.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{fixedfirsttraining2026,
title = {Adaptive-Depth Is a Finishing Stage},
author = {Andrei Piterbarg and Sean McLeish and Dhruv Pai and Benjamin Keigwin},
year = {2026},
eprint = {2610.14252},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.14252}
}