NextTransformer IAAI logoIAAIInteractive & Active AI

NextTransformer Blog / Arena v1

How Opus Found AttnRes-Canon: The Discovery Trace of an Architecture Search

Abstract

We ran an auto-research experiment on NextTransformer's Arena v1: Claude Opus 5.5 designed and screened Transformer architectures on its own, with people approving plans and budgets. The work had three steps: early exploration, workflow design with three rounds of search, and a final full training run. Each search candidate trained for only 2,000 updates, about 4% of a full run; the winner then trained from scratch for 49.152B tokens under the fixed protocol and was ranked by CORE-22. The result, AttnRes-Canon, combines Full AttnRes with Canon layers at four positions: after full training it scores 35.99 on CORE, 2.0 points above the search's starting point Block AttnRes, with validation BPB 0.0024 lower, and it ranks first on both among the results published on the leaderboard.

01 / Exploration

The Agent's Early Exploration

The brief for Opus was simple: under the Arena v1 rules, improve the Transformer architecture so that, after full training, it beats the methods already on the leaderboard on CORE-22. There are only four rules:

01

Fixed

The training data, the length of training (46,875 updates) and the compute per token (FLOPs, within ±1%) are locked, the same for every method.

02

Free to change

Only the model architecture. Each idea is a “candidate”, trained under the same conditions.

03

How it is scored

After full training, by CORE-22: the mean score over 22 tasks, higher is better. Validation BPB is reported too; lower is better.

04

What the search may look at

Only training loss and BPB on the dev split (bits per byte: the average number of bits needed to predict each byte; lower is better).

The hard part is cost. One full run takes more than 1,400 GPU-hours, so the search cannot train every idea to the end. Opus had to train each candidate for a short stretch (a short run, below) and use its dev BPB to guess how the full run would end. If that ruler is off, everything after it is guesswork. So before changing any model, it spent about 150 GPU-hours on exploration, testing whether short runs rank methods correctly.

That depends on how training is cut short. A full run keeps the learning rate constant at first and lowers it gradually in the later part; this plan is the learning-rate schedule. There are two ways to shorten it (Figure 1): compressing squeezes the whole schedule into 512 updates, while a prefix leaves the schedule alone and takes only the first stretch of the formal run.

Figure 1. Two ways to cut training short.

Opus tried compression first: it trained 11 methods that already had full-training results for 512 updates each on the compressed schedule (about 1% of a full run) and compared the order with their final ranking (Figure 2).

Figure 2. Short-run rank versus final rank.

The compressed schedule ranks poorly. The Spearman correlation between the two rankings was only 0.40 (1 means the same order, 0 means no relation), far below the 0.85 bar set in advance. The errors were systematic: all four looped methods (the same layers applied repeatedly) fell, and all four linear-attention methods rose. Lowering the learning rate early helped the fast-converging looped models and cut off the slow-starting linear-attention ones before they got going.

A prefix does much better. Checked against the published full-training curves of the same 11 methods (Figure 3), a prefix of about 500 updates has a Spearman of 0.72; at 2,000 updates it reaches 0.92, above the bar.

Figure 3. Ranking reliability by prefix length.

That settled the ruler: leave the learning-rate schedule alone and train the first 2,000 updates of the formal schedule. Update 512 only removes candidates that clearly failed to train; ranks and wins are decided at update 2,000.

02 / Workflow

Designing the Search Workflow

With the ruler set, Opus wrote the findings into a set of search rules (Figure 4, Table 1): at most three rounds and 600 GPU-hours in total.

Figure 4. The search workflow.
Table 1. Exploration findings and the rules that follow.
Finding of the explorationRule in the workflow
A compressed schedule ranks poorly (Spearman 0.40)Do not compress the schedule; take a prefix of the formal schedule
The first 500 updates do not rank well enough (Spearman 0.72; the bar is 0.85)Update 512 is only a loose floor: stop a candidate early only if it is more than 0.030 behind the Dense Baseline
The first 2,000 updates rank well enough (Spearman 0.92)Rank candidates and decide wins at update 2,000
Repeating the same run gives a slightly different result (call the gap δ)Measure δ first; a candidate wins only if it is below the best baseline by max(2δ, 0.001)

Two more conventions applied in practice:

  • The winning line is set by Block AttnRes: the search first trained two baselines: the Arena's Dense Baseline and Block AttnRes, the strongest method on the leaderboard at the time (listed there as AttnRes). Training the Dense Baseline again with the same random seed showed that two runs differ by about 0.0004 at update 2,000 (δ). So a candidate's dev BPB had to be more than 0.001 below Block AttnRes to win.
  • One change at a time, stacked over rounds: each candidate changes one thing relative to its parent (the model it modifies) and states in advance a hypothesis an experiment can refute. A later round may take an earlier candidate as the parent, so changes stack round by round.

03 / Search

Three Search Rounds and Full Training

3.1Overview and Search Path

Starting from Block AttnRes, the search trained 12 candidates over three rounds, and 4 of them beat the winning line (Figure 5).

Figure 5. The search path.

None of the five round-1 candidates actually crossed the line. The bold path in the figure kept the two changes that came closest, Full AttnRes and Canon-A/C; round 2 extended and combined them, and round 3 merged them into the final AttnRes-Canon.

The research trace replays the whole session on a time axis: every instruction, every reply and each experiment from launch to result.

3.2Round 1: Single Changes and an Early Mirage

The starting point, Block AttnRes, splits its 23 layers into 8 groups and lets every layer read back the outputs of the earlier groups. Round 1 tried five single changes on top of it: three from recent papers, and two that adjust the model's depth-to-width ratio and where its parameters go (Table 2).

Table 2. The five single changes of round 1 (dev BPB relative to Block AttnRes; lower is better).
ChangeHypothesis written at the timeUpdate 512Update 2,000
Full AttnRes: 23 groups instead of 8, so every layer can read any earlier layer on its own [1]Finer early information can be retrieved−0.0010−0.0008
Canon-A/C: a small width-4 convolution at the input of attention (A) and of the MLP (C) [2]Mixes information from neighbouring tokens at very low cost−0.0006−0.0006
Gate: a learned switch (sigmoid gate) on the output of each attention head [3]Adjusts each head's output by the input and removes useless attention−0.0027+0.0006
Deep-narrow: the narrowest FFN the protocol allows, 27 layers instead of 23At equal compute, a few more layers pay off more than a wider FFN+0.0076+0.0027
Value embedding in every layerAbout 0.55B more parameters at almost no extra compute+0.0022+0.0007

At update 2,000 no candidate crossed the winning line of −0.0010; Full AttnRes (−0.0008) and Canon-A/C (−0.0006) came closest. Gate's large early lead was a mirage: at update 512 it was 0.0027 lower, first in the field; by about update 1,000 the lead was gone, and at update 2,000 it was 0.0006 behind (Figure 6).

Figure 6. Round 1: dev BPB relative to Block AttnRes (lower is better).

3.3Round 2: Combinations and Marginal Contributions

As soon as round 1 reached update 512, Opus chose the round-2 combinations from the ranking at that moment and added two MoE variants at equal compute (the FFN is replaced by several small experts, and each token uses only a few of them) [4] (Table 3). The rules fixed when wins are decided, not when the next direction is chosen, so the early leader Gate took half of the combination slots.

Table 3. Round 2: dev BPB relative to Block AttnRes (lower is better).
CandidateUpdate 512Update 2,000Result
Canon-A–D: beyond A/C, Canon also on q/k/v (B) and on the MLP hidden layer (D)−0.0018−0.0021Won
Full + Canon-A/C−0.0020−0.0018Won
Full + Gate−0.0025−0.0010Won (just over the line)
Gate + Canon-A/C−0.0038−0.0001Did not win
MoE: 16 half-width experts, 4 chosen per token+0.0007+0.0016Did not win
MoE: 8 experts, 2 chosen per token+0.0055+0.0063Did not win

Three candidates crossed the line in this round, but the early order flipped even more completely than in round 1: the top four at update 512 finished in exactly the reverse order at update 2,000 (Figure 7). Gate again scrambled the early ranking: at update 512, every model with Gate was 0.0014 to 0.0031 below the same model without it; by update 2,000 that gap was back near zero in every case.

Figure 7. Round 2: dev BPB relative to Block AttnRes (lower is better).

Before training, Opus expected nearly the same from Full + Canon-A/C and Gate + Canon-A/C: the two changes act on different parts of the model, so their gains should add up. The first crossed the line comfortably and the second barely moved. The reason shows when each change is taken on its own: how much further does the same change lower dev BPB when it is added to different parents (Figure 8, Table 4)?

Figure 8. Marginal effect of each change on different models.
Table 4. Marginal effect of each change at update 2,000 (change in dev BPB).
Change addedOn Block AttnResOn Full AttnResOn Canon-A/COn Gate
Full AttnRes−0.0008—−0.0012−0.0016
Canon-A/C−0.0006−0.0010—−0.0007
Gate+0.0006−0.0002+0.0005—
  • Full AttnRes and Canon give steady gains: on every parent tested, each lowered dev BPB at update 2,000.
  • Gate gives no net gain: its three results fall on both sides of zero and are about as large as the random difference between repeated runs (about 0.0004); Full + Gate crossed the line because of Full AttnRes.

“Different parts of the model, so the gains add up” has a condition: each change has to work on its own. Gate did not, so combining it with other changes added nothing, and round 3 dropped it.

3.4Round 3: Shared-Parent Comparison and the Final Model

After the lesson of the first two rounds, Opus did not go by the update-512 ranking in round 3; it waited until round 2 was well into its later updates. By then only Full AttnRes and Canon still stood. Two round-2 candidates happened to share Canon-A/C as their parent and each took one step from it: one switched to Full AttnRes, and the other added positions B and D to become Canon-A–D. Compared on that shared parent, both were about 0.002 below Canon-A/C by update 1,536 (Figure 9). The remaining budget covered only one more candidate, so Opus did not wait for Canon-A–D's last few hundred updates and decided to merge the two. That is AttnRes-Canon.

Figure 9. Dev BPB relative to the Canon-A/C parent.

AttnRes-Canon reached a dev BPB of 0.7965 at update 2,000, 0.0038 below Block AttnRes and first in the whole search (Table 5).

Table 5. Dev BPB at update 2,000 along the two directions (Δ: difference from Block AttnRes).
Depth \ CanonNo CanonCanon-A/CCanon-A–D
Block AttnRes (8 groups)0.8003baselinestep time ×1.000.7997Δ −0.0006step time ×1.100.7982Δ −0.0021step time ×1.47
Full AttnRes (23 groups)0.7995Δ −0.0008step time ×1.190.7985Δ −0.0018step time ×1.300.7965Δ −0.0038step time ×1.66

The two components work in two directions (Figure 10). In Block AttnRes a layer can read only the sum of each earlier group's outputs, like reading only chapter summaries; Full AttnRes removes the groups, so a layer can read any earlier layer on its own. A Canon layer is a small width-4 convolution: while processing the current token it also mixes in the three tokens before it. One fetches information along depth, the other mixes neighbouring tokens along the sequence, and together they gain a little more than the sum of their separate gains (Table 5).

Figure 10. Where Full AttnRes and the Canon layers act.

3.5Results of Full Training

The search sees only the very beginning of training, before the learning rate starts to fall. Whether the lead lasts to the end has to be tested by full training. We ported AttnRes-Canon into a formal plugin and trained it from scratch for 49.152B tokens. Evaluation uses two metrics: validation BPB, and CORE-22 (the mean score over 22 tasks after subtracting the level of random guessing, multiplied by 100 here as on the leaderboard; higher is better). AttnRes-Canon ranks first among the 38 published results on both CORE and validation BPB; second on both is mHC, a cross-layer connection method (Table 6, Figure 11).

Table 6. Results after full training.
MethodCOREValidation BPB
Dense Baseline33.100.6672
Block AttnRes34.000.6630
mHC35.330.6617
AttnRes-Canon35.990.6605
Figure 11. CORE and validation BPB after full training.

The validation BPB lead holds from start to finish, but more than half of it is lost in the later part. The gap to Block AttnRes was 0.0037 at update 2,000, grew to nearly 0.006 while the learning rate was constant, was mostly clawed back once the learning rate began to fall, and ended at 0.0024 (Figure 12, Table 7).

Figure 12. Differences from Block AttnRes over full training.
Table 7. Validation BPB gaps at three points of full training.
ComparisonUpdate 2,000Constant learning rate (mean of 10K–16K)End
AttnRes-Canon vs Block AttnRes−0.0037−0.0058−0.0024
Dense Baseline + Gate vs Dense Baseline−0.0012−0.0004+0.0002

The second row of Table 7 is a clearer case: another contributor fully trained the Dense Baseline with Gate added; it led by 0.0012 at update 2,000, enough to cross this search's winning line of 0.001, and finished 0.0002 behind. How much of a lead built at a constant learning rate survives is known only when training is finished.

04 / Takeaways

Summary and Outlook

4.1Takeaways from Arena v1

What decided this search was whether early mirages were seen through: short runs have to be long enough to filter out changes that lead only at the start, and before changes are combined, each must be shown to work on its own on a shared parent.

The three stages came to 2,135 GPU-hours, two thirds of it for the final full run (Table 8).

Table 8. Compute across the three stages (GPU-hours).
StageWhat it coversUsedShare
Early explorationChecking the research environment; testing whether short runs can rank1527%
Workflow design and searchTwo baselines, the noise measurement and 12 candidates over three rounds55026%
Final trainingA full run of AttnRes-Canon under the protocol1,43367%
Total2,135100%

What Opus got right:

  1. Calibrated the ruler before searching: once it found that the compressed schedule misranks methods systematically, it switched to the first 2,000 updates of the formal schedule.
  2. Changed one thing at a time: each candidate changed one thing relative to its parent and came with a refutable hypothesis written in advance, which let round 2 separate out each change's contribution.
  3. Waited for a longer signal before combining: in round 3 it no longer went by the update-512 ranking; it compared on the shared parent Canon-A/C up to update 1,536 before merging.
  4. Used files as memory: over four days its conversation context was compacted 7 times, and each time the candidate notes and selection records it had written to files let it carry on.

Where it fell short:

  1. A plan without a reminder: it meant to decide round 3 at update 1,536 but set reminders only at updates 512 and 2,000, stopped in between, and resumed only when the operator prompted it.
  2. Its prior judgement was off: equal-compute MoE, the idea it rated highest, was tried in two versions and both finished last.
  3. It mixed up two kinds of random variation at first: it treated the difference between repeated runs with the same random seed as the difference between seeds, and never ran a second seed.
  4. It looked at quality, not speed: the selection rules had no limit on time per update, so the chosen AttnRes-Canon takes 1.66 times as long per update as Block AttnRes (Table 5).

4.2A Preview of Arena v2

In v1, people still approved each stage, watched the budget, decided when to stop and ported the plugin. Arena v2 plans four changes that hand the whole process to the agent and the Gym:

  • A sealed container: all research runs inside an isolated container through the Gym's interface; the agent cannot see the formal validation set or CORE-22.
  • A fixed compute allowance: each session gets 1.96×10²⁰ FLOPs (roughly 20 baseline runs of 2,000 updates in the v1 search); training, evaluation and failed retries are all charged to it.
  • Experiments designed by the model: which model to start from, which controls to run, how many candidates to try, how long to train each and when to drop one are all up to the model; no update counts or rounds are preset.
  • One file to submit: after a single model file is submitted, the Gym starts the formal full run itself, with no stage-by-stage approval or manual porting.

As planned, Opus, GLM and DeepSeek will each run one independent study on the same 1.96×10²⁰ FLOPs allowance, and the architectures they find will be compared under the same full-training protocol.

This experiment completed architecture exploration within one session. The next step is to test self-improvement across sessions (RSI): write lessons such as “decide automatically at the planned point” and “check each change on a shared parent” into the next workflow, and compare it directly with the frozen old workflow at the same compute, to see whether the agent can turn the mistakes it made into a steadier search.

References

  1. Attention Residuals. Kimi Team. arXiv preprint, 2026. arXiv:2603.15031
  2. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. Zeyuan Allen-Zhu. arXiv preprint, 2025. arXiv:2512.17351
  3. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. Zihan Qiu et al. arXiv preprint, 2025. arXiv:2505.06708
  4. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. Noam Shazeer et al. ICLR, 2017. arXiv:1701.06538