IAAIDreamer
Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
Depth-recurrent attention mixtures combine sequence attention, attention along depth and sparse expert attention, relieving the fixed hidden-size bottleneck of many-step latent reasoning and decoupling scaling dimensions in depth-recurrent models.
Inside the method
[ Sequence attention + depth attention + sparse expert attention ] × r
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Depth recurrence with attention over earlier depths
- KV / state strategy
- Attention over states from earlier depths
Reading note
Needs 2–8× fewer training tokens for the same accuracy as FLOP-, parameter- and memory-matched baselines, and shows 2–11× more expert selection diversity than standard MoEs.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{dreamer2026,
title = {Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves},
author = {Jonas Knupp and Jan Hendrik Metzen and Jeremias Bohn and Georg Groh and Kristian Kersting},
year = {2026},
eprint = {2601.21582},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2601.21582}
}