Contribute to the T3 sweep
Pick a family and depth in the T3 explorer and download its design manifest. For example, dense-T3-L20 has 20 layers, width 1,280 and 10 heads. Its leading-term estimate is 29,433 updates, or 30.86B tokens. These are planning figures until checked against the actual run.
Model reference code
Section titled “Model reference code”The Contribute tab opens the reference links, implementation work and coordination issue for any of the 16 T3 designs. Start with dense-T3-L20 for the shared training and evaluation checks.
| Family | Start in the repository | T3 work to contribute |
|---|---|---|
| Vanilla Dense | Full-attention backbone and no-VE benchmark configuration | Use the selected width and depth with full causal attention. Verify cost, checkpoint resume and the evaluation schedule. |
| Vanilla MoE | RoutedFFN computes a normalized top-k mixture with one shared ReLU² expert. Megatron-Core provides a distributed implementation for comparison. |
Keep a dense first layer. Connect expert dispatch for 256 routed experts, eight selected per token and one shared expert. Pilot the balancing rule and measure cost. |
| Hybrid, as KDA–DSA | KdaDsaGroup composes three KDA mixers and one DSA mixer, with a feed-forward block after each mixer. |
Repeat the group to the selected depth over the same RoutedFFN recipe as Vanilla MoE, keeping a dense first layer. Include the indexer loss and verify training FLOPs for the first 512 updates and the following sparse phase separately. |
| Looped, as fixed recursive | Shared-core execution and the benchmark configuration | Reuse the same core weights on both passes, including the experts, over the same RoutedFFN recipe as Vanilla MoE with a dense first layer. Adapt the benchmark’s attention windows and value embeddings to full attention with VE off. Charge both executions, including backward. |
Try the references on CPU
Section titled “Try the references on CPU”From the checkout with the project dependencies installed:
NANOCHAT_DTYPE=float32 OMP_NUM_THREADS=1 python -m nexttransformer.scaling.reference_blocksThe command runs a forward and backward pass on synthetic inputs for the existing Dense and fixed recursive benchmark models and for the three expert families. MoE uses eight small experts and selects two; Looped and Hybrid place the same routed block above their own structure, keeping one dense layer at the bottom of the stack. KDA–DSA exercises both sides of the burn-in boundary. A formal T3 run still needs a matching trainer, verified cost and measured throughput.
Coordinate a contribution
Section titled “Coordinate a contribution”- Open a T3 coordination issue with the cell ID, intended contribution and availability. Verify the implementation and cost proof with the maintainers before booking compute.
- For implementation work, match the 2K protocol, check the kernels, data stream, memory, resume behavior and FLOPs for each training phase. Publish estimated and measured throughput separately.
- For a formal run, pin the approved implementation, configuration, cost proof and data hashes. Preserve seed 42 and report every scheduled evaluation. Keep the final validation BPB, full CORE-22, actual tokens and cumulative training FLOPs together with checkpoint and provenance links.
Analysis of the fits, audits and evaluations of released checkpoints are also welcome. Later phases open when their own specifications are published.