IAAIWISE
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Finds that attention routing in recurrent-depth models stabilizes well before hidden states do. WISE uses full attention in early recurrent steps to discover a block-structured working set, then reuses that support in later steps while keeping attention weights and refinement dynamic; no training is needed.
Inside the method
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed recurrence in two stages
- KV / state strategy
- Later steps attend only to the discovered working set
Reading note
Largely preserves full-attention performance on multi-hop QA, with better trade-offs at longer contexts; a sparse implementation gives up to 1.76× late-step attention speedup over FlashAttention at 4K context.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{wise2026,
title = {Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models},
author = {Ke Wan and Chen Chen},
year = {2026},
eprint = {2609.27373},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27373}
}