IAAIContinuous Depth Batching
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
The first efficient serving method for depth-adaptive looped models: new batches are formed between loop steps, looped and non-looped parts are scheduled separately, looped KV caches are managed, and early exits are predicted so batches can be prepared asynchronously.
Inside the method
Loop step k: batch of tokens still looping ‖ exited tokens → non-looped tail
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- Per-token adaptive depth at serving time
- KV / state strategy
- Looped KV-cache management across exits
Reading note
Realizes up to 99% of the estimated maximum speedup on Ouro 1.4B and Huginn 3.5B. Fully looped architectures suit adaptive inference best, since large non-looped parts complicate scheduling.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{continuousdepthbatching2026,
title = {Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching},
author = {Kristian Schwethelm and Daniel Rueckert and Georgios Kaissis},
year = {2026},
eprint = {2608.09444},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.09444}
}