IAAIHyperloop Transformer
Hyperloop Transformers
Organizes a looped Transformer into begin, middle and end blocks, loops only the middle block, and adds hyper-connections after each loop to expand the residual stream into matrix-valued streams at minimal extra cost.
Inside the method
Begin block → [ Middle block ] × R (hyper-connections after each loop) → End block
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed recurrence
- KV / state strategy
- See paper
Reading note
Performs well against depth-matched Transformer and mHC baselines with about 50% fewer parameters, and the advantage persists after post-training weight quantization.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{hyperloop2026,
title = {Hyperloop Transformers},
author = {Abbas Zeitoun and Lucas Torroba-Hennigen and Yoon Kim},
year = {2026},
eprint = {2604.21254},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2604.21254}
}