← Back to the collection
Adaptive depth ·
MoEUT
MoEUT: Mixture-of-Experts Universal Transformers
Combines the Universal Transformer with mixture-of-experts layers to study shared depth and sparse capacity.
Inside the method
[ Shared attention + MoE ] × R
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- See paper
- KV / state strategy
- See paper
Reading note
Code and checkpoint availability have not been audited for this entry.
Sources checked 2026-09-15. This catalog does not imply independent reproduction.
Cite this work
@misc{moeut2024,
title = {MoEUT: Mixture-of-Experts Universal Transformers},
author = {Róbert Csordás and Kazuki Irie and Jürgen Schmidhuber and Christopher Potts and Christopher D. Manning},
year = {2024},
eprint = {2405.16039},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2405.16039}
}