IAAILoopMTP
LoopMTP: A looped transformer guided by latent multi-token prediction
Links looping to multi-token prediction: a model that loops T times should anticipate T future tokens, so the hidden state at loop t is softly aligned with the embedding of the token t steps ahead, with a gate preserving useful information across iterations.
Inside the method
[ Stack ] × T, loop t ↔ embedding of token t steps ahead
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed loops (stable up to 15)
- KV / state strategy
- See paper
Reading note
Improves average accuracy by up to 8.1% relative over the non-looped baseline, with stable training up to 15 loops.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopmtp2026,
title = {LoopMTP: A looped transformer guided by latent multi-token prediction},
author = {Behzad Shomali and Markus Frey and David Berghaus and Joachim Koehler and Mehdi Ali},
year = {2026},
eprint = {2608.03624},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.03624}
}