IAAIAccordion
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
A mixture-of-experts model that serves five inference budgets through fixed-prefix layer looping, with the smallest budget equal to the unlooped model. Pretraining samples one budget per micro-batch, and SFT and RL with budget-matched multi-teacher distillation preserve every budget.
Inside the method
Unlooped path → fixed-prefix layer looping for budgets 1…5
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- Five predefined execution paths chosen at inference
- KV / state strategy
- See paper
Reading note
After 3.6T pretraining tokens, the deepest budget raises Accordion-6B-A0.5B's macro-average over 37 tasks by 3.01 points; the middle budget captures 89% of the gain with half the extra layer executions.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{accordion2026,
title = {Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping},
author = {Yudong Zhang and Wei Ding and Kaili Zheng and Yuchen Ding and Zhenyu Zhang},
year = {2026},
eprint = {2610.32492},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.32492}
}