NextTransformer Blog / Arena v1
How Opus Found AttnRes-Canon: The Discovery Trace of an Architecture Search
We ran an auto-research experiment on NextTransformer's Arena v1: Claude Opus 5.5 designed and screened Transformer architectures on its own, with people approving plans and budgets. The work had three steps: early exploration, workflow design with three rounds of search, and a final full training run. Each search candidate trained for only 2,000 updates, about 4% of a full run; the winner then trained from scratch for 49.152B tokens under the fixed protocol and was ranked by CORE-22. The result, AttnRes-Canon, combines Full AttnRes with Canon layers at four positions: after full training it scores 35.99 on CORE, 2.0 points above the search's starting point Block AttnRes, with validation BPB 0.0024 lower, and it ranks first on both among the results published on the leaderboard.
01 / Exploration
The Agent's Early Exploration
The brief for Opus was simple: under the Arena v1 rules, improve the Transformer architecture so that, after full training, it beats the methods already on the leaderboard on CORE-22. There are only four rules:
Fixed
The training data, the length of training (46,875 updates) and the compute per token (FLOPs, within ±1%) are locked, the same for every method.
Free to change
Only the model architecture. Each idea is a “candidate”, trained under the same conditions.
How it is scored
After full training, by CORE-22: the mean score over 22 tasks, higher is better. Validation BPB is reported too; lower is better.
What the search may look at
Only training loss and BPB on the dev split (bits per byte: the average number of bits needed to predict each byte; lower is better).
The hard part is cost. One full run takes more than 1,400 GPU-hours, so the search cannot train every idea to the end. Opus had to train each candidate for a short stretch (a short run, below) and use its dev BPB to guess how the full run would end. If that ruler is off, everything after it is guesswork. So before changing any model, it spent about 150 GPU-hours on exploration, testing whether short runs rank methods correctly.
That depends on how training is cut short. A full run keeps the learning rate constant at first and lowers it gradually in the later part; this plan is the learning-rate schedule. There are two ways to shorten it (Figure 1): compressing squeezes the whole schedule into 512 updates, while a prefix leaves the schedule alone and takes only the first stretch of the formal run.
Opus tried compression first: it trained 11 methods that already had full-training results for 512 updates each on the compressed schedule (about 1% of a full run) and compared the order with their final ranking (Figure 2).
The compressed schedule ranks poorly. The Spearman correlation between the two rankings was only 0.40 (1 means the same order, 0 means no relation), far below the 0.85 bar set in advance. The errors were systematic: all four looped methods (the same layers applied repeatedly) fell, and all four linear-attention methods rose. Lowering the learning rate early helped the fast-converging looped models and cut off the slow-starting linear-attention ones before they got going.
A prefix does much better. Checked against the published full-training curves of the same 11 methods (Figure 3), a prefix of about 500 updates has a Spearman of 0.72; at 2,000 updates it reaches 0.92, above the bar.
That settled the ruler: leave the learning-rate schedule alone and train the first 2,000 updates of the formal schedule. Update 512 only removes candidates that clearly failed to train; ranks and wins are decided at update 2,000.
02 / Workflow
Designing the Search Workflow
With the ruler set, Opus wrote the findings into a set of search rules (Figure 4, Table 1): at most three rounds and 600 GPU-hours in total.
| Finding of the exploration | Rule in the workflow |
|---|---|
| A compressed schedule ranks poorly (Spearman 0.40) | Do not compress the schedule; take a prefix of the formal schedule |
| The first 500 updates do not rank well enough (Spearman 0.72; the bar is 0.85) | Update 512 is only a loose floor: stop a candidate early only if it is more than 0.030 behind the Dense Baseline |
| The first 2,000 updates rank well enough (Spearman 0.92) | Rank candidates and decide wins at update 2,000 |
| Repeating the same run gives a slightly different result (call the gap δ) | Measure δ first; a candidate wins only if it is below the best baseline by max(2δ, 0.001) |
Two more conventions applied in practice:
- The winning line is set by Block AttnRes: the search first trained two baselines: the Arena's Dense Baseline and Block AttnRes, the strongest method on the leaderboard at the time (listed there as AttnRes). Training the Dense Baseline again with the same random seed showed that two runs differ by about 0.0004 at update 2,000 (δ). So a candidate's dev BPB had to be more than 0.001 below Block AttnRes to win.
- One change at a time, stacked over rounds: each candidate changes one thing relative to its parent (the model it modifies) and states in advance a hypothesis an experiment can refute. A later round may take an earlier candidate as the parent, so changes stack round by round.
03 / Search
Three Search Rounds and Full Training
3.1Overview and Search Path
Starting from Block AttnRes, the search trained 12 candidates over three rounds, and 4 of them beat the winning line (Figure 5).
None of the five round-1 candidates actually crossed the line. The bold path in the figure kept the two changes that came closest, Full AttnRes and Canon-A/C; round 2 extended and combined them, and round 3 merged them into the final AttnRes-Canon.
The research trace replays the whole session on a time axis: every instruction, every reply and each experiment from launch to result.
3.2Round 1: Single Changes and an Early Mirage
The starting point, Block AttnRes, splits its 23 layers into 8 groups and lets every layer read back the outputs of the earlier groups. Round 1 tried five single changes on top of it: three from recent papers, and two that adjust the model's depth-to-width ratio and where its parameters go (Table 2).
| Change | Hypothesis written at the time | Update 512 | Update 2,000 |
|---|---|---|---|
| Full AttnRes: 23 groups instead of 8, so every layer can read any earlier layer on its own [1] | Finer early information can be retrieved | −0.0010 | −0.0008 |
| Canon-A/C: a small width-4 convolution at the input of attention (A) and of the MLP (C) [2] | Mixes information from neighbouring tokens at very low cost | −0.0006 | −0.0006 |
| Gate: a learned switch (sigmoid gate) on the output of each attention head [3] | Adjusts each head's output by the input and removes useless attention | −0.0027 | +0.0006 |
| Deep-narrow: the narrowest FFN the protocol allows, 27 layers instead of 23 | At equal compute, a few more layers pay off more than a wider FFN | +0.0076 | +0.0027 |
| Value embedding in every layer | About 0.55B more parameters at almost no extra compute | +0.0022 | +0.0007 |
At update 2,000 no candidate crossed the winning line of −0.0010; Full AttnRes (−0.0008) and Canon-A/C (−0.0006) came closest. Gate's large early lead was a mirage: at update 512 it was 0.0027 lower, first in the field; by about update 1,000 the lead was gone, and at update 2,000 it was 0.0006 behind (Figure 6).
3.3Round 2: Combinations and Marginal Contributions
As soon as round 1 reached update 512, Opus chose the round-2 combinations from the ranking at that moment and added two MoE variants at equal compute (the FFN is replaced by several small experts, and each token uses only a few of them) [4] (Table 3). The rules fixed when wins are decided, not when the next direction is chosen, so the early leader Gate took half of the combination slots.
| Candidate | Update 512 | Update 2,000 | Result |
|---|---|---|---|
| Canon-A–D: beyond A/C, Canon also on q/k/v (B) and on the MLP hidden layer (D) | −0.0018 | −0.0021 | Won |
| Full + Canon-A/C | −0.0020 | −0.0018 | Won |
| Full + Gate | −0.0025 | −0.0010 | Won (just over the line) |
| Gate + Canon-A/C | −0.0038 | −0.0001 | Did not win |
| MoE: 16 half-width experts, 4 chosen per token | +0.0007 | +0.0016 | Did not win |
| MoE: 8 experts, 2 chosen per token | +0.0055 | +0.0063 | Did not win |
Three candidates crossed the line in this round, but the early order flipped even more completely than in round 1: the top four at update 512 finished in exactly the reverse order at update 2,000 (Figure 7). Gate again scrambled the early ranking: at update 512, every model with Gate was 0.0014 to 0.0031 below the same model without it; by update 2,000 that gap was back near zero in every case.
Before training, Opus expected nearly the same from Full + Canon-A/C and Gate + Canon-A/C: the two changes act on different parts of the model, so their gains should add up. The first crossed the line comfortably and the second barely moved. The reason shows when each change is taken on its own: how much further does the same change lower dev BPB when it is added to different parents (Figure 8, Table 4)?
| Change added | On Block AttnRes | On Full AttnRes | On Canon-A/C | On Gate |
|---|---|---|---|---|
| Full AttnRes | −0.0008 | — | −0.0012 | −0.0016 |
| Canon-A/C | −0.0006 | −0.0010 | — | −0.0007 |
| Gate | +0.0006 | −0.0002 | +0.0005 | — |
- Full AttnRes and Canon give steady gains: on every parent tested, each lowered dev BPB at update 2,000.
- Gate gives no net gain: its three results fall on both sides of zero and are about as large as the random difference between repeated runs (about 0.0004); Full + Gate crossed the line because of Full AttnRes.
“Different parts of the model, so the gains add up” has a condition: each change has to work on its own. Gate did not, so combining it with other changes added nothing, and round 3 dropped it.
3.4Round 3: Shared-Parent Comparison and the Final Model
After the lesson of the first two rounds, Opus did not go by the update-512 ranking in round 3; it waited until round 2 was well into its later updates. By then only Full AttnRes and Canon still stood. Two round-2 candidates happened to share Canon-A/C as their parent and each took one step from it: one switched to Full AttnRes, and the other added positions B and D to become Canon-A–D. Compared on that shared parent, both were about 0.002 below Canon-A/C by update 1,536 (Figure 9). The remaining budget covered only one more candidate, so Opus did not wait for Canon-A–D's last few hundred updates and decided to merge the two. That is AttnRes-Canon.
AttnRes-Canon reached a dev BPB of 0.7965 at update 2,000, 0.0038 below Block AttnRes and first in the whole search (Table 5).
| Depth \ Canon | No Canon | Canon-A/C | Canon-A–D |
|---|---|---|---|
| Block AttnRes (8 groups) | 0.8003baselinestep time ×1.00 | 0.7997Δ −0.0006step time ×1.10 | 0.7982Δ −0.0021step time ×1.47 |
| Full AttnRes (23 groups) | 0.7995Δ −0.0008step time ×1.19 | 0.7985Δ −0.0018step time ×1.30 | 0.7965Δ −0.0038step time ×1.66 |
The two components work in two directions (Figure 10). In Block AttnRes a layer can read only the sum of each earlier group's outputs, like reading only chapter summaries; Full AttnRes removes the groups, so a layer can read any earlier layer on its own. A Canon layer is a small width-4 convolution: while processing the current token it also mixes in the three tokens before it. One fetches information along depth, the other mixes neighbouring tokens along the sequence, and together they gain a little more than the sum of their separate gains (Table 5).
3.5Results of Full Training
The search sees only the very beginning of training, before the learning rate starts to fall. Whether the lead lasts to the end has to be tested by full training. We ported AttnRes-Canon into a formal plugin and trained it from scratch for 49.152B tokens. Evaluation uses two metrics: validation BPB, and CORE-22 (the mean score over 22 tasks after subtracting the level of random guessing, multiplied by 100 here as on the leaderboard; higher is better). AttnRes-Canon ranks first among the 38 published results on both CORE and validation BPB; second on both is mHC, a cross-layer connection method (Table 6, Figure 11).
| Method | CORE | Validation BPB |
|---|---|---|
| Dense Baseline | 33.10 | 0.6672 |
| Block AttnRes | 34.00 | 0.6630 |
| mHC | 35.33 | 0.6617 |
| AttnRes-Canon | 35.99 | 0.6605 |
The validation BPB lead holds from start to finish, but more than half of it is lost in the later part. The gap to Block AttnRes was 0.0037 at update 2,000, grew to nearly 0.006 while the learning rate was constant, was mostly clawed back once the learning rate began to fall, and ended at 0.0024 (Figure 12, Table 7).
| Comparison | Update 2,000 | Constant learning rate (mean of 10K–16K) | End |
|---|---|---|---|
| AttnRes-Canon vs Block AttnRes | −0.0037 | −0.0058 | −0.0024 |
| Dense Baseline + Gate vs Dense Baseline | −0.0012 | −0.0004 | +0.0002 |
The second row of Table 7 is a clearer case: another contributor fully trained the Dense Baseline with Gate added; it led by 0.0012 at update 2,000, enough to cross this search's winning line of 0.001, and finished 0.0002 behind. How much of a lead built at a constant learning rate survives is known only when training is finished.
04 / Takeaways
Summary and Outlook
4.1Takeaways from Arena v1
What decided this search was whether early mirages were seen through: short runs have to be long enough to filter out changes that lead only at the start, and before changes are combined, each must be shown to work on its own on a shared parent.
The three stages came to 2,135 GPU-hours, two thirds of it for the final full run (Table 8).
| Stage | What it covers | Used | Share |
|---|---|---|---|
| Early exploration | Checking the research environment; testing whether short runs can rank | 152 | 7% |
| Workflow design and search | Two baselines, the noise measurement and 12 candidates over three rounds | 550 | 26% |
| Final training | A full run of AttnRes-Canon under the protocol | 1,433 | 67% |
| Total | 2,135 | 100% |
What Opus got right:
- Calibrated the ruler before searching: once it found that the compressed schedule misranks methods systematically, it switched to the first 2,000 updates of the formal schedule.
- Changed one thing at a time: each candidate changed one thing relative to its parent and came with a refutable hypothesis written in advance, which let round 2 separate out each change's contribution.
- Waited for a longer signal before combining: in round 3 it no longer went by the update-512 ranking; it compared on the shared parent Canon-A/C up to update 1,536 before merging.
- Used files as memory: over four days its conversation context was compacted 7 times, and each time the candidate notes and selection records it had written to files let it carry on.
Where it fell short:
- A plan without a reminder: it meant to decide round 3 at update 1,536 but set reminders only at updates 512 and 2,000, stopped in between, and resumed only when the operator prompted it.
- Its prior judgement was off: equal-compute MoE, the idea it rated highest, was tried in two versions and both finished last.
- It mixed up two kinds of random variation at first: it treated the difference between repeated runs with the same random seed as the difference between seeds, and never ran a second seed.
- It looked at quality, not speed: the selection rules had no limit on time per update, so the chosen AttnRes-Canon takes 1.66 times as long per update as Block AttnRes (Table 5).
4.2A Preview of Arena v2
In v1, people still approved each stage, watched the budget, decided when to stop and ported the plugin. Arena v2 plans four changes that hand the whole process to the agent and the Gym:
- A sealed container: all research runs inside an isolated container through the Gym's interface; the agent cannot see the formal validation set or CORE-22.
- A fixed compute allowance: each session gets 1.96×10²⁰ FLOPs (roughly 20 baseline runs of 2,000 updates in the v1 search); training, evaluation and failed retries are all charged to it.
- Experiments designed by the model: which model to start from, which controls to run, how many candidates to try, how long to train each and when to drop one are all up to the model; no update counts or rounds are preset.
- One file to submit: after a single model file is submitted, the Gym starts the formal full run itself, with no stage-by-stage approval or manual porting.
As planned, Opus, GLM and DeepSeek will each run one independent study on the same 1.96×10²⁰ FLOPs allowance, and the architectures they find will be compared under the same full-training protocol.
This experiment completed architecture exploration within one session. The next step is to test self-improvement across sessions (RSI): write lessons such as “decide automatically at the planned point” and “check each change on a shared parent” into the next workflow, and compare it directly with the frozen old workflow at the same compute, to see whether the agent can turn the mistakes it made into a steadier search.
The result is on the leaderboard and the Arena page of the Observatory. Formal run record, disclosure card and method code: run record · disclosure card · method plugin
References
- Attention Residuals. Kimi Team. arXiv preprint, 2026. arXiv:2603.15031
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. Zeyuan Allen-Zhu. arXiv preprint, 2025. arXiv:2512.17351
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. Zihan Qiu et al. arXiv preprint, 2025. arXiv:2505.06708
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. Noam Shazeer et al. ICLR, 2017. arXiv:1701.06538