IAAILooped Latent Attention
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Shows the loop-indexed KV cache of looped models traces a short low-rank trajectory per token, layer and head. Looped Latent Attention is a post-training codec that stores compact K/V latents and reconstructs loop-specific K/V only when attention reads them.
Inside the method
Per-head latent across loops → reconstruct K/V for loop r on read
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed loops of the base model
- KV / state strategy
- Compressed cross-loop K/V latents (SVD-initialized, distilled)
Reading note
At matched cache budget it beats MLA-style head compression, cross-layer sharing, KV quantization and final-loop reuse. On one H200, Ouro-1.4B batch capacity at 4K context rises from 32 to 768 sequences at 21.3× compression; transfers to Huginn-3.5B.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopedlatentattention2026,
title = {Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers},
author = {James O' Neill and Fergal Reid},
year = {2026},
eprint = {2607.15456},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.15456}
}