Model zoo
NextTransformer has 36 model cards in seven families. Every method runs in the same shell and is calibrated to the same training compute. Open a method in the observatory for its architecture diagram, curves and model card; “registered” methods have a card and a configuration but no published run.
| Family | Method | Paper | Result |
|---|---|---|---|
| Baseline | Dense Baseline (dense_full) |
Attention Is All You Need | published |
| Looped | Universal Transformer · ACT (ut_act) |
Universal Transformers | published |
| Looped | Fixed Recursive (fixed_recursive) |
Fixed Recursive · shared-core control baseline | published |
| Looped | Huginn · Recurrent-depth (huginn) |
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach | published |
| Looped | Ouro · LoopLM (ouro) |
Scaling Latent Reasoning via Looped Language Models | published |
| Looped | DeepLoop (deeploop) |
DeepLoop · v2 | published |
| Looped | Mixture-of-Recursions · MoR (mor) |
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation | published |
| Looped | LoopFormer-Fixed2 (loopformer) |
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation | published |
| Looped | Circle Loop (circle) |
Circle Loop · method specification | published |
| Sparse | DeepSeek Sparse Attention · DSA (dsa) |
DeepSeek-V3.2-Exp · DeepSeek Sparse Attention | published |
| Sparse | Qwen Sparse Attention · QSA (qsa) |
Qwen3.8-Next technical report · §2.1.2 QSA | published |
| Sparse | Native Sparse Attention · NSA (nsa) |
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention | published |
| Sparse | SeerAttention-style (seer) |
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs | published |
| Sparse | Mixture of Block Attention · MoBA (moba) |
MoBA: Mixture of Block Attention for Long-Context LLMs | published |
| Linear | DeltaNet (deltanet) |
Parallelizing Linear Transformers with the Delta Rule over Sequence Length | published |
| Linear | Gated Slot Attention · GSA + QK-norm (gsa_qknorm) |
Gated Slot Attention for Efficient Linear-Time Sequence Modeling | published |
| Linear | Gated Linear Attention · GLA (gla) |
Gated Linear Attention Transformers with Hardware-Efficient Training | published |
| Linear | Kimi Delta Attention · KDA (kda) |
Kimi Linear: An Expressive, Efficient Attention Architecture | published |
| Linear | RetNet · Retention (retnet) |
Retentive Network: A Successor to Transformer for Large Language Models | published |
| Linear | ELU-Norm (elu_qknorm) |
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | published |
| Linear | Gated DeltaNet · GDN (gdn) |
Gated Delta Networks: Improving Mamba2 with Delta Rule | published |
| Residual | Standard Residual (residual_standard) |
Identity Mappings in Deep Residual Networks | published |
| Residual | Block Attention Residuals · AttnRes (attnres) |
Attention Residuals | published |
| Residual | Hyper-Connections · HC (hc) |
Hyper-Connections | published |
| Residual | Manifold-Constrained Hyper-Connections · mHC (mhc) |
mHC: Manifold-Constrained Hyper-Connections | published |
| Hybrid | Hybrid GDN + MHA (gdn_hybrid) |
Gated Delta Networks: Improving Mamba2 with Delta Rule | published |
| Hybrid | Hybrid GDN + SWA (gdn_hybrid_swa) |
Gated Delta Networks: Improving Mamba2 with Delta Rule | published |
| Hybrid | Hybrid DeltaNet + MHA (deltanet_hybrid) |
Parallelizing Linear Transformers with the Delta Rule over Sequence Length | coming soon |
| Hybrid | Hybrid KDA + MLA-NoPE (kda_hybrid) |
Kimi Linear: An Expressive, Efficient Attention Architecture | published |
| Hybrid | Hybrid GDN + QSA (qsa_hybrid) |
Qwen3.8-Next technical report · §2.1.2 QSA | coming soon |
| Hybrid | Hybrid MoBA + MHA (moba_hybrid) |
MoBA: Mixture of Block Attention for Long-Context LLMs | coming soon |
| Others | SSSL (sssl) |
nanochat · GPT d24 implementation | published |
| Others | Multi-Query Attention · MQA (mqa) |
Fast Transformer Decoding: One Write-Head is All You Need | published |
| Others | Grouped-Query Attention · GQA (gqa) |
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints | published |
| Others | Multi-head Latent Attention · MLA + QK-norm (mla_qknorm) |
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model | published |
| Others | MLA-NoPE + QK-norm (mla_nope_qknorm) |
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model | published |
See the configuration reference for training configurations. Add your own as a plugin.