IAAIAsymLoop
Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder
Separates context building from iterative refinement: a single-pass causal encoder writes fixed per-layer global KV caches, and a weight-tied recurrent decoder attends jointly to them and to local KV from its evolving state.
Inside the method
Causal encoder (once, Global KV) → [ Weight-tied decoder, Local KV ] × R
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed recurrence at matched effective depth
- KV / state strategy
- Fixed global KV from the encoder plus local KV per iteration, under one softmax
Reading note
From 60M to 1B on FineWeb-Edu it beats Full Loop at matched effective depth and tokens; at 1B the seven-task zero-shot average rises from 49.58% to 50.85%. Benefits extend to an OLMo-style backbone and MoE.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{asymloop2026,
title = {Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder},
author = {Cheng Liang and Tengfei Zhang and Wenjie DU and Jianyu Wu and Changlian Ma and Shuchen Pu and Ya Zhang and Yanfeng Wang and Zhouhan Lin},
year = {2026},
eprint = {2610.28489},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28489}
}