| 基线 |
Dense Baseline (dense_full) |
Attention Is All You Need |
已发布 |
| 循环 |
Universal Transformer · ACT (ut_act) |
Universal Transformers |
已发布 |
| 循环 |
Fixed Recursive (fixed_recursive) |
Fixed Recursive · shared-core control baseline |
已发布 |
| 循环 |
Huginn · Recurrent-depth (huginn) |
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach |
已发布 |
| 循环 |
Ouro · LoopLM (ouro) |
Scaling Latent Reasoning via Looped Language Models |
已发布 |
| 循环 |
DeepLoop (deeploop) |
DeepLoop · v2 |
已发布 |
| 循环 |
Mixture-of-Recursions · MoR (mor) |
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation |
已发布 |
| 循环 |
LoopFormer-Fixed2 (loopformer) |
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation |
已发布 |
| 循环 |
Circle Loop (circle) |
Circle Loop · method specification |
已发布 |
| 稀疏 |
DeepSeek Sparse Attention · DSA (dsa) |
DeepSeek-V3.2-Exp · DeepSeek Sparse Attention |
已发布 |
| 稀疏 |
Qwen Sparse Attention · QSA (qsa) |
Qwen3.8-Next technical report · §2.1.2 QSA |
已发布 |
| 稀疏 |
Native Sparse Attention · NSA (nsa) |
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention |
已发布 |
| 稀疏 |
SeerAttention-style (seer) |
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs |
已发布 |
| 稀疏 |
Mixture of Block Attention · MoBA (moba) |
MoBA: Mixture of Block Attention for Long-Context LLMs |
已发布 |
| 线性 |
DeltaNet (deltanet) |
Parallelizing Linear Transformers with the Delta Rule over Sequence Length |
已发布 |
| 线性 |
Gated Slot Attention · GSA + QK-norm (gsa_qknorm) |
Gated Slot Attention for Efficient Linear-Time Sequence Modeling |
已发布 |
| 线性 |
Gated Linear Attention · GLA (gla) |
Gated Linear Attention Transformers with Hardware-Efficient Training |
已发布 |
| 线性 |
Kimi Delta Attention · KDA (kda) |
Kimi Linear: An Expressive, Efficient Attention Architecture |
已发布 |
| 线性 |
RetNet · Retention (retnet) |
Retentive Network: A Successor to Transformer for Large Language Models |
已发布 |
| 线性 |
ELU-Norm (elu_qknorm) |
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention |
已发布 |
| 线性 |
Gated DeltaNet · GDN (gdn) |
Gated Delta Networks: Improving Mamba2 with Delta Rule |
已发布 |
| 残差 |
Standard Residual (residual_standard) |
Identity Mappings in Deep Residual Networks |
已发布 |
| 残差 |
Block Attention Residuals · AttnRes (attnres) |
Attention Residuals |
已发布 |
| 残差 |
Hyper-Connections · HC (hc) |
Hyper-Connections |
已发布 |
| 残差 |
Manifold-Constrained Hyper-Connections · mHC (mhc) |
mHC: Manifold-Constrained Hyper-Connections |
已发布 |
| 混合 |
Hybrid GDN + MHA (gdn_hybrid) |
Gated Delta Networks: Improving Mamba2 with Delta Rule |
已发布 |
| 混合 |
Hybrid GDN + SWA (gdn_hybrid_swa) |
Gated Delta Networks: Improving Mamba2 with Delta Rule |
已发布 |
| 混合 |
Hybrid DeltaNet + MHA (deltanet_hybrid) |
Parallelizing Linear Transformers with the Delta Rule over Sequence Length |
待发布 |
| 混合 |
Hybrid KDA + MLA-NoPE (kda_hybrid) |
Kimi Linear: An Expressive, Efficient Attention Architecture |
已发布 |
| 混合 |
Hybrid GDN + QSA (qsa_hybrid) |
Qwen3.8-Next technical report · §2.1.2 QSA |
待发布 |
| 混合 |
Hybrid MoBA + MHA (moba_hybrid) |
MoBA: Mixture of Block Attention for Long-Context LLMs |
待发布 |
| 其他 |
SSSL (sssl) |
nanochat · GPT d24 implementation |
已发布 |
| 其他 |
Multi-Query Attention · MQA (mqa) |
Fast Transformer Decoding: One Write-Head is All You Need |
已发布 |
| 其他 |
Grouped-Query Attention · GQA (gqa) |
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints |
已发布 |
| 其他 |
Multi-head Latent Attention · MLA + QK-norm (mla_qknorm) |
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model |
已发布 |
| 其他 |
MLA-NoPE + QK-norm (mla_nope_qknorm) |
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model |
已发布 |
