Claude models compared cost vs quality at every effort level

Data analysis · aggregated from 356 sources · 1724 benchmarks · 7845 measurements
GitHub
Alexandre Gensse

What each recent Claude model actually costs, at every effort level, and the quality it delivers — fused from public benchmark measurements for every (model, effort) pair.

The best value today is Haiku 5.5 at high effort — 2.0× cheaper than the price trend for its quality (expected score 59.3 % on the benchmark panel).

Quality is the expected score across the whole benchmark panel, not one benchmark: the type of task can move it, see what moves everything.

Measurements
356 sources · 1724 benchmarks · 7845 measurements
Updated
19 Nov 2026
👑 Best overall
Haiku 5.5 · High Cost 2.8× · Score 59.3 %
197
Grunt work
Haiku 5.5 · Low Cost 1.2× · Score 52.8 %
122
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Everyday tasks
Haiku 5.5 · Medium Cost 1.7× · Score 56.7 %
188
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Advanced reasoning
Fable 5.2 · High Cost 128× · Score 74.9 %
137
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Cutting-Edge thinking
Fable 5.2 · Max Cost 438× · Score 76.5 %
65
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.

Quality vs. cost of every Claude model, at each effort level

One point per couple: a model at one configuration (thinking off, minimal, low → max, thinking on); no line joins a model's couples. Only the couples of the Pareto frontier are labelled (model · effort), where there is room; hover or tap any point for its model, effort, cost and score. Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward. Haiku 4.5 lives only in the matrix and Pareto.

Best Claude model for the price — the Pareto frontier

The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated; the frontier couples are labelled (model · effort) where there is room — hover or tap any point for its model, effort, cost and score. No line joins a model's couples. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.

Value — distance to the price trend

The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 21 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.90, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.

Model · effortcostexpected scorevalue index
trend = 100
Haiku 5.5 · High2.8×59.3 %197
Haiku 5.5 · Medium1.7×56.7 %188
Haiku 5.5 · xHigh5.1×61.9 %180
Fable 5.2 · Medium67×73.2 %165
Fable 5.2 · High128×74.9 %137
Fable 5.2 · Low58×71.8 %135
Opus 5.5 · High53×71.4 %135
Opus 5.5 · Medium42×70.1 %124
Sonnet 5.5 · xHigh45×70.4 %123
Haiku 5.5 · Low1.2×52.8 %122
Fable 5.2 · xHigh199×75.5 %106
Sonnet 5.5 · High23×66.4 %99
Haiku 5.5 · Max15×63.9 %92
Fable 5.2 · Max438×76.5 %65
Haiku 5.5 · Thinking off1.0×44.3 %23

Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.

Best value by task tier — what to actually run

Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.

Target 45.8 % – Grunt work
Haiku 5.5 · Low Cost 1.2× · Score 52.8 %
122
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Target 55.9 % – Everyday tasks
Haiku 5.5 · Medium Cost 1.7× · Score 56.7 %
188
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Target 66.5 % – Advanced reasoning
Fable 5.2 · High Cost 128× · Score 74.9 %
137
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Target 75.3 % – Cutting-Edge thinking
Fable 5.2 · Max Cost 438× · Score 76.5 %
65
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.
👑 Best overall
Haiku 5.5 · High
Cost 2.8× · Score 59.3 % · Value index 197

Picked as the frontier couple that sits furthest below the price trend : it costs 2.0× less than the trend charges for its quality — or, read on the other axis, it scores 3.4 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.

Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — Claude effort levels compared

Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.

Modelthinking offlowmediumhighxhighmaxExpected score
top effort
Fable 5.2—
5850–66
6758–77
128113–145
199173–229
438385–497
76.5 %
Opus 5.5—
2927–31
4240–44
5350–55
9488–100
236220–254
73.9 %
Sonnet 5.5—
1513–16
1615–18
2321–26
4540–51
150131–171
72.6 %
Haiku 5.5
1.00.85–1.2
1.21.1–1.3
1.71.6–1.9
2.82.5–3.1
5.14.5–5.7
1513–16
63.9 %

"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. A model measured at a single setting (solo : no effort parameter, see the measurement sources) fills one merged cell. A thinking off column appears when a model is measured with its thinking switched off, a minimal one for the lowest level some models have below low; each is a configuration of its own, read from its own measurements. n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).

Value-index matrix — every model and effort against the price trend

The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.

Modelthinking offlowmediumhighxhighmax
Fable 5.2—
135102–180
165125–219
137106–177
10680–139
6550–84
Opus 5.5—
6656–77
124109–141
135119–152
117101–135
5749–66
Sonnet 5.5—
3428–41
5445–64
9984–116
123102–148
6452–78
Haiku 5.5
2313–40
12299–152
188155–228
197165–236
180146–223
9276–111

Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.

What moves everything — four factors behind the numbers

The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.

Task type

Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.

Complexity

The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).

Generation

Each release buys the same quality for less — Sonnet 5.5 matches Sonnet 5's max quality at its own low effort for 10× less per task, Fable 5.2 matches Fable 5.1's max at its own low for 6.7× less, Opus 5.5 matches Opus 5's max at its own medium for 4.7× less, Fable 5.1 matches Fable 5's max at its own medium for 3.3× less, and across families Opus 5.5 matches Fable 5.1's max at its own medium for 8.1× less; Sonnet 5 is the exception: a third cheaper per token than Sonnet 4.6 but verbose, it costs about the same per task (0.9× at the same effort).

Cache & harness

Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.

Data Sources

Every source below measured at least two (model, effort) couples on the same task; 10115 cost and 23548 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.

SourceActual config (harness · effort)TypeLinked couples (same task)
AA CritPt
artificialanalysis
AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓sweepOpus 5.5·low · Opus 5.5·xhigh · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 4.7·max · Opus 5.5·medium · Opus 5.5·high · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
AA GDP-PDF
artificialanalysis
AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓sweepFable 5.1·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5·low · Opus 5·high · Opus 5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·high · Sonnet 5·max
AA GDPval v2
artificialanalysis
AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓sweepSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·low · Fable 5.1·xhigh · Fable 5.1·high · Opus 5.5·xhigh · Opus 5.5·medium · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Haiku 5.5·max · Fable 5.2·max · Fable 5.2·xhigh · Fable 5.2·high · Fable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Opus 5·low · Sonnet 5·low · Opus 5·medium · Sonnet 5·medium · Opus 5·high · Sonnet 5·high · Opus 5·xhigh · Sonnet 5·xhigh · Opus 4.7·max
AA Harvey LAB
artificialanalysis
AA model payload · Harvey's Legal Agent Bench (AA's own run, separate from Vals's HLAB), score-only, 8 models × ladder + 4 at max ✓sweepSonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·max
AA HLE
artificialanalysis
AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓sweepFable 5.1·low · Opus 5.5·low · Opus 5.5·xhigh · Opus 5.5·medium · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·max
AA MLCR
artificialanalysis
AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓sweepSonnet 5.5·max · Opus 5.5·max · Haiku 5.5·max
AA Omniscience
artificialanalysis
AA model payload · Omniscience accuracy (not the penalised index) ✓sweepSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·low · Fable 5.1·xhigh · Fable 5.1·high · Opus 5.5·xhigh · Opus 5.5·medium · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 5·high
AA SciCode
artificialanalysis
AA model payload · SciCode, 4 models × ladder + 3 at max ✓sweepFable 5.1·low · Fable 5.1·medium · Opus 5.5·low · Fable 5.1·xhigh · Opus 5.5·xhigh · Opus 5.5·medium · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 5.5·high · Opus 5·low · Fable 5.1·high · Fable 5.1·max · Fable 5·max · Sonnet 5·low · Sonnet 5·high
AA TB 4.0
artificialanalysis
AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓sweepOpus 5.5·xhigh · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
AA Terminal-Bench-Science
artificialanalysis
AA model payload · Terminal-Bench-Science 0.1 (AA harness, separate from Vals's board), score-only, 4 models × ladder + 2 at max ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·xhigh · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max · Opus 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Haiku 5.5·xhigh
AA-Briefcase
artificialanalysis
AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·xhigh · Opus 5.5·xhigh · Fable 5.1·low · Fable 5.1·medium · Haiku 5.5·max · Sonnet 5.5·high · Sonnet 5.5·max · Opus 5.5·high · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 5·low · Sonnet 5·low · Opus 5·medium · Sonnet 5·medium · Opus 5·high · Sonnet 5·high · Opus 5·xhigh · Sonnet 5·xhigh
AA-LCR
artificialanalysis
AA model payload · long-context reasoning, input-dominated cost ✓sweepFable 5.1·medium · Opus 5.5·xhigh · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·medium · Sonnet 5·high
Agents on Rails · features
rubyonrails
rubyonrails.org/ai + rails/ai-evals runs · 20 tickets ×3, miniswen; Opus 5 & Fable 5.1 high→max, Opus 5.5 default ✓sweepOpus 5.5·medium · Fable 5.1·high · Fable 5.1·max · Opus 5·high · Opus 5·max
AI4AI-Bench
arxiv-2608.20318
arXiv 2608.20318 · 10 ML-research repos, measured exploration spend, 2 models × ladder ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
Angular shopping cart, Opus 5.5 effort ladder (OctAcademy video)
youtube-octacademy
Claude Code v2.1.280; same CLAUDE.md and prompt; fresh branch per run; one run per level; 10-point checklist scored by a fresh Claude session; 2026-10-16; read 2026-10-18 ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
Anthropic docs: max_tokens cap vs cost (internal repo tasks) (16k cap)
anthropic
~135 internal repository tasks ('cap measurement'); Opus 5.5 at its default (medium), Fable 5.1 at its default (high); cost = whole-dollar bar labels, digitised from cost-intel-max-tokens-saving-opus-5-5.png; read 2026-10-01 ✓; 16k max_tokens cap only; score onlysweepOpus 5.5·medium · Fable 5.1·high
Anthropic docs: max_tokens cap vs cost (internal repo tasks) (64k cap)
anthropic
~135 internal repository tasks ('cap measurement'); Opus 5.5 at its default (medium), Fable 5.1 at its default (high); cost = whole-dollar bar labels, digitised from cost-intel-max-tokens-saving-opus-5-5.png; read 2026-10-01 ✓; 64k max_tokens cap only; score onlysweepOpus 5.5·medium · Fable 5.1·high
ARC-AGI-1
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓sweepHaiku 4.5·nothink · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·max · Sonnet 4.6·high · Opus 5·high · Opus 5·max · Fable 5·max · Fable 5·xhigh · Fable 5·high · Fable 5·medium · Fable 5·low · Fable 5.1·max · Fable 5.1·xhigh · Fable 5.1·high · Fable 5.1·medium · Fable 5.1·low · Opus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Haiku 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low · Sonnet 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Fable 5.2·xhigh · Fable 5.2·high · Fable 5.2·medium · Fable 5.2·low
ARC-AGI-2
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓sweepHaiku 4.5·nothink · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Sonnet 4.6·high · Opus 5·high · Opus 5·max · Fable 5·max · Fable 5·xhigh · Fable 5·high · Fable 5·medium · Fable 5·low · Fable 5.1·max · Fable 5.1·xhigh · Fable 5.1·high · Fable 5.1·medium · Fable 5.1·low · Opus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Haiku 5.5·xhigh · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low · Sonnet 5.5·xhigh · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Fable 5.2·xhigh · Fable 5.2·high · Fable 5.2·medium · Fable 5.2·low
arXiv 2609.13463 · TRAIL root-cause attribution
arxiv-2609.13463
Root-cause attribution of agent failures on TRAIL, Continual Search judge in the Claude Agent SDK, turn 4 · weighted F1 and cumulative mean cost per trajectory (USD), read from the vector coordinates of figures 4–5 (they agree with the numbers the text states); Opus 4.8 and Sonnet 5 at low/high/max (figure 5), Fable 5 at default settings (figure 4, whose Opus and Sonnet points are figure 5's high runs); read 30 Sep 2026 ✓sweepOpus 4.8·low · Opus 4.8·high · Opus 4.8·max · Sonnet 5·low · Sonnet 5·high · Sonnet 5·max
ArXivMath no-tools
anthropic
Opus 5.5 card p182 · ArXivMath Aug 2026, 57 problems, 3 models × low→max, $ cost, scores printed ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max
ArXivMath tools
anthropic
Opus 5.5 card p183 · ArXivMath Aug 2026 with code sandbox, 3 models × low→max, scores printed ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max
AutomationB. 5
anthropic
Opus 5 card p180 · Zapier, linear-$ chart; Fable=Mythos 5, effort inferred ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max
BenchCAD V2C
anthropic
Opus 5.5 card p205 · Vision2Code with tools, voxel IoU, 4 models × 5 efforts; effort inferred from cost order ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
BenchCAD V2C −tools
anthropic
Sonnet 5.5 card p129 · tools disabled, 4 models × 5, separate regime kept out of the effort grid ✓sweepSonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max
BrowseComp 10M
anthropic
Opus 5 card p160 · 10M budget, sweep low→max; Fable=Mythos 5, $ cost ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
BrowseComp 500-cut (Anthropic docs)
anthropic
Fable 5, 1-3 runs per setting ✓sweepFable 5·low · Fable 5·medium · Fable 5·high
bug-hunt-bench
github-bughuntbench
Claude Code 2.1.251 · 105 planted bugs, blind judge, 3 models × ladder ✓sweepFable 5.1·max · Fable 5.1·high · Fable 5.1·low · Fable 5·max · Fable 5·high · Opus 5·max · Opus 5·xhigh · Opus 5·high · Opus 5·medium · Opus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Opus 4.8·high · Opus 4.8·max · Sonnet 5·high · Sonnet 5·max · Fable 5.1·medium · Fable 5.1·xhigh · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Fable 5.2·max
BullshitBench v2
github-bullshitbench
petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓sweepOpus 4.8·nothink · Opus 4.8·xhigh · Sonnet 4.6·high · Sonnet 4.6·nothink · Fable 5.1·low · Opus 4.7·nothink · Fable 5·xhigh · Opus 5·xhigh · Sonnet 5·low · Sonnet 5·max · Opus 5·low · Fable 5.1·max · Fable 5·low · Opus 5.5·low · Opus 5.5·max · Haiku 4.5·nothink · Opus 4.7·max · Sonnet 5.5·low · Haiku 5.5·max · Haiku 5.5·low · Sonnet 5.5·max
Buoy: Haiku 5.5 default vs high (256 food-ordering-app jobs)
buoy.gg
256 jobs in Buoy's own test app on iOS Simulators, 1 run each; score = jobs done right %; cost per job from tokens x list price; read 2026-10-15 ✓sweepHaiku 5.5·high
CFG k3s DevOps
computingforgeeks
computingforgeeks Sonnet 5.5 review · 3 tasks × 3 runs, raw API single file; score = k3s deploy + HTTP 200 pass rate of the K8s task, cost summed over the 3 tasks ✓sweepSonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5·high
Chartography +tools
anthropic
Fable 5.1 card p186 · 5 models × sweep low→max, $ cost; Opus 4.8 series joined from Opus 5 card p171 ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max
Chartography −tools
anthropic
Fable 5.1 card p186 · tools disabled — separate regime, kept out of the effort grid ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max
Chartography regraded
anthropic
Opus 5.5 card p202 · same transcripts as chartogt, re-graded scores — quality only, costs stay in chartogt ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
claude-effort-bench
github-effort-bench
Claude Code 2.1.206 · 3 multi-file tasks × 5 reps, Opus 5 & Sonnet 5 × medium→xhigh, CLI cost (Sonnet ×2/3), deterministic verifiers; ~30 % silent no-op runs scored 0 ✓sweepOpus 5·medium · Opus 5·high · Opus 5·xhigh · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
claudecode_lab · invoice check
note-claudecode_lab
note.com 29 Sep · Claude Code 2.1.284 -p, Sonnet 5.5 and Opus 5.5 × low/medium/high/max, 7/7 everywhere (saturated), cost from Claude Code ✓sweepSonnet 5.5·medium · Sonnet 5.5·max
Cookbook cost optimization
anthropic
Anthropic cookbook cost_optimization.ipynb · 10-claim agent ×2 trials, Opus 5 & Sonnet 5 low→high, Haiku 4.5; list-price cost from usage ✓sweepOpus 5·high · Opus 5·medium · Opus 5·low · Sonnet 5·high · Sonnet 5·medium · Sonnet 5·low
Corpus defect sweep (Anthropic docs)
anthropic
21.6M-token corpus, F1, per episode ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high
CursorBench
anthropic
Sonnet 5 card p118 · 5 models × sweep low→max, $ cost, scores printed ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·low · Sonnet 4.6·high · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max
CursorBench 3.2
cursorbench
Cursor 3.2 · matched effort, $ cost ✓ (vendor benchmark)sweepOpus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·xhigh · Fable 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Opus 5·low · Opus 5·medium · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
CursorBench 4.0
cursorbench
cursor.com payload · v4.0 long-horizon suite, Opus 5/5.5, Fable 5.1, Sonnet 5/5.5 × low→max, measured tokens × list incl. cache ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Fable 5.2·max · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·medium · Fable 5.2·low
CyberQ security short answer
cyberq
Opus 5.5 ladder + Opus 5, no score ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·high
CyberQ structured extraction
cyberq
Opus 5.5 ladder + Opus 5, no score ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·high
Databricks · Claude Code
databricks
Opus 4.8 high/xhigh/max + Sonnet 5/4.6, Haiku 4.5 ✓sweepOpus 4.8·high · Opus 4.8·max · Opus 4.8·xhigh
DeepResearch Bench II (Anthropic docs)
anthropic
50-task subset, 33-task basis, customer-billed ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5·low · Fable 5·medium · Fable 5·high
DeepSearchQA
anthropic
Opus 4.7 card p200 · Opus4.7 vs Sonnet4.6, sweep low→max ✓sweepOpus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 4.6·low · Sonnet 4.6·high
DeepSearchQA (O5)
anthropic
Opus 5 card p161 · 980k budget, 4 models × sweep low→max, $ cost ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
DeepWideSearch (Anthropic docs)
anthropic
Fable 5, 220 questions, 3 runs ✓sweepFable 5·low · Fable 5·medium · Fable 5·high
DRACO
anthropic
Fable 5 card p271 · Fable=Mythos 5, sweep low→max, $ cost ✓sweepOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max
DRACO (F5.1)
anthropic
Fable 5.1 card p179 · 980k budget, 3 models × sweep low→max, $ cost, scores printed ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh
DRACO (O5.5)
anthropic
Opus 5.5 card p187 · 980k budget, 3 models × low→max, $ cost, scores printed; a new run, not the F5.1-card one ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
DRACO (Opus 5)
anthropic
Opus 5 card p162 · 980k budget, sweep low→max; Fable=Mythos 5, $ cost ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
FrontierCode main
cognition
Cognition leaderboard JSON · v1.1 main (100 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.7·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
FrontierCode v1
cognition
Sonnet 5 card p117 · 4 models × sweep low→max, $ cost, scores printed ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·xhigh
FrontierCode-D
anthropic
Fable 5 card p257 · Fable=Mythos 5, sweep low→max, $ cost ✓sweepOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max
FrontierCode-Ext
cognition
Cognition leaderboard JSON · v1.1 Extended (150 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.7·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
FrontierFinance
arxiv-2608.11683
arXiv 2608.11683 Fig. 9 · Finance Agent v2 harness, Opus 4.8 low→xhigh, API cost/query (vector-read) ✓sweepOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh
GDPval (Anthropic docs)
anthropic
Fable 5, 1 run, cost incl. grading ✓sweepFable 5·low · Fable 5·medium · Fable 5·high
George Liu · Opus 5 vs 5.5
ai.georgeliu
Claude Code 10-prompt suite · low→max · IFEval /9 ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
Haiku 5.5 launch page: GDPval-AA v2.1, accuracy vs cost
anthropic
anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
Haiku 5.5 launch page: Humanity's Last Exam (no tools), accuracy vs cost
anthropic
anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
Haiku 5.5 launch page: OSWorld 2.1 (offline subset), accuracy vs cost
anthropic
anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
Haiku 5.5 system card: HealthBench by effort, 5 levels
anthropic
1 run/level below max, max = mean of 5 runs; read 2026-10-08 ✓sweepHaiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·low
Haiku 5.5 system card: HealthBench Professional (length-adjusted) by effort, 5 levels
anthropic
5 runs/level except 13 repeat runs at max on a different day (mean 64.2% vs reported 64.8%, see confound); read 2026-10-08 ✓sweepHaiku 5.5·max · Haiku 5.5·high
Haiku 5.5 system card: PhysicianBench by effort, 5 levels, pass rate
anthropic
5 attempts/task at each level; read 2026-10-08 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
haiku-thinking-bench: does thinking-off make Haiku 5.5 skip a needed handoff (24 cases x 10 runs, 80 handoff runs)
github-haiku-thinking-bench
17tayyy/haiku-thinking-bench README (Oct 2026); mechanical grading; score = handoffs made correctly of 80 (100 - missed%); cost = cost per conversation (tokens x list price, no caching); read 2026-10-15 ✓sweepHaiku 4.5·nothink · Haiku 5.5·nothink · Haiku 5.5·low · Haiku 5.5·medium
harnesseval
github-harnesseval
Claude Code · 6 PRs common to every cell, CLI-metered cost ✓sweepFable 5.1·high · Opus 5·high · Sonnet 5·high
HealthBench Pro (adj.)
anthropic
Sonnet 5.5 card p138 · length-adjusted, Sonnet 5.5 low→max + 3 at max, score-only ✓sweepSonnet 5.5·max · Opus 5.5·max · Fable 5.1·max
HLE (Opus 5)
anthropic
Opus 5 card p157 · HLE tools, 4 models × sweep low→max, $ cost ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
HLE no-tools (F5.1)
anthropic
Fable 5.1 card p178 · HLE without tools, 3 models × sweep low→max, $ cost, scores printed ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·medium · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
HLE no-tools (S55)
anthropic
Sonnet 5.5 card p120 · 980k budget, 3 models × low→max, $ cost (perfect caching), scores printed ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
HLE sweep
anthropic
Opus 4.8 card p203 · HLE tools, sweep low→max ✓sweepOpus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max
HLE tools (F5.1)
anthropic
Fable 5.1 card p177 · HLE with tools, 3 models × sweep low→max, $ cost, scores printed ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·medium
HLE tools (O5.5)
anthropic
Opus 5.5 card p185 · HLE with tools, 3 models × sweep low→max, $ cost, scores printed ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
ibragim.dev · Claude
ibragim-dev
Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Claude Code 2.1.281, effort per row, pass@1 and $/task; board of 28 Sep ✓sweepOpus 5.5·high · Sonnet 5·high
ibragim.dev · Pi
ibragim-dev
Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Pi 0.85.1, effort per row, pass@1 and $/task; board of 28 Sep ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5·high · Sonnet 5·high
Internal coding, 370 repo tasks (Anthropic docs)
anthropic
plain API agent, customer-billed per attempt ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high
jev-effort
github-jev-effort
Claude Code 2.1.280 · Opus 5.5 baseline arm, high vs max fixed effort, 6 coding tasks x2 runs ✓sweepOpus 5.5·high · Opus 5.5·max
kayben plan review
github-kaybenleroll
claude -p no tools, CC 2.1.282 · 13 seeded defects, Opus 5.5 & Sonnet 5 × medium/high/xhigh, n=10, effort verified in transcripts ✓sweepOpus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
kayben plan review f51
github-kaybenleroll
same doc, cap10 prompt · Fable 5.1 low→max, n=5 ✓sweepOpus 5.5·max · Sonnet 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max
Lisbon bike-rental website, Sonnet 5.5 effort ladder (Mahe Delich video)
youtube-mahe-delich
Claude Code /cost; same prompt; empty folder; one try per level; 2026-09-30; read 2026-10-18 ✓sweepSonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
llm-regression 2026-10-02 Opus 5.5 high vs xhigh (pass rate %)
nikoamoretti-llm-regression
nikoamoretti/llm-regression data.json: 2026-10-02 first live runs, product suite, Claude Code, high (32 attempts) vs xhigh (28 attempts, 3 infra_fail); strict hidden-test pass rate over graded attempts; read 2026-10-20 ✓sweepOpus 5.5·xhigh · Opus 5.5·high
Low-poly forest in Blender, Haiku 5.5 effort ladder (DopeShit video)
youtube-dopeshit
Claude Code, Haiku 5.5 effort set per run; author grades 4 criteria (10 each) = /40; 2026-10-10; read 2026-10-18 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
m_izumo Opus 5.5 review sub-agents on a personal project (15 sections)
qiita-m_izumo
Claude Code sub-agents (20 x high; 22 x xhigh); defects verified by main session; coverage = share of verified real defects found by one reviewer; read 2026-10-15 ✓sweepOpus 5.5·high · Opus 5.5·xhigh
MARROWLIGHT one-page site, Opus 5.5 effort ladder (The Useful Stack video)
youtube-usefulstack
Claude Code 2.1.285 --effort; auto-scored /100; redone runs only; one run per setting; 2026-09-30; read 2026-10-18 ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh
MERC core-10
github-merc-bench
Claude Code -p, no tools · 10 graded tasks, 6 models × low→max, n=2; usage × list (5m cache-write); scores 94–98/98 → cost only ✓sweepFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·xhigh
nnakapa #35 Copilot
zenn-nnakapa
GitHub Copilot CLI 1.0.79 · same 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, billed cost ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
nnakapa #36 Copilot
zenn-nnakapa
GitHub Copilot CLI 1.0.80 · same 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, billed cost ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
nnakapa QCD #20
zenn-nnakapa
Claude Code 2.1.157 · 3 DB tasks × 10 runs, Opus 4.7 & Opus 4.8 × low→xhigh ✓sweepOpus 4.7·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh
nnakapa QCD #24
zenn-nnakapa
Claude Code 2.1.170 · 3 DB tasks × 10 runs, Fable 5 & Opus 4.8 × low→xhigh ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh
nnakapa QCD #29
zenn-nnakapa
Claude Code 2.1.220 · 3 DB tasks × 10 runs, Opus 5, Opus 4.8 & Fable 5 × low→xhigh, CLI-reconciled cost, hidden-test gate ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh
nnakapa QCD #34
zenn-nnakapa
Claude Code 2.1.226 · 3 DB tasks × 10 runs, Sonnet 5 & Opus 5 × low→xhigh, CLI-reconciled cost ($2/$10 Sonnet), hidden-test gate ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh
nnakapa QCD #35 CC
zenn-nnakapa
Claude Code 2.1.229 (1M ctx) · 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, CLI-reconciled cost ($2/$10) ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
nnakapa QCD #37
zenn-nnakapa
Claude Code 2.1.258 · 3 DB tasks × 10 runs, Opus 5 & Fable 5.1 × low→xhigh, CLI-reconciled cost, hidden-test gate ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh
ObviousBench 0.2
benchmarklist
inspect-ai · 144 items ×3 epochs, 3 models × full ladder, measured tokens ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Haiku 5.5·nothink · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Opus 4.8·nothink · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 4.6·nothink · Sonnet 4.6·low · Sonnet 4.6·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
Occam bench, no plugin
github-occam
Claude Code headless · 18 procedural tasks, hidden verifier · Sonnet 5.5 & Opus 5.5 at max/medium + Haiku 4.5 ✓sweepOpus 5.5·max · Opus 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·medium
Opus 5.5 AoE2 build
github-barty-opus55
Claude Code 2.1.281 · one long game-build prompt, Opus 5.5 × low→max, 1 run, CLI cost, no score → cost only ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·high · Sonnet 5.5·max
OSWorld
anthropic
Anthropic/AA · sweep low→max ✓sweepSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 4.6·low · Sonnet 4.6·high
OSWorld 2.0
anthropic
Opus 5 card p174 · price vs perf, 4 models; effort inferred from cost order ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
OSWorld 2.0 (F5.1)
anthropic
Fable 5.1 card p190 · partial-credit price/perf, 3 models; effort inferred from cost order ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max
OSWorld 2.0 board (Opus 5)
osworld-v2-board
osworld-v2.xlang.ai official JSON · v2.1 full 108 tasks, batched, 500 steps, Opus 5 low→max, output tokens/task, no cost ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
OSWorld 2.0 board v08.08 (Opus 5)
osworld-v2-board
osworld-v2.xlang.ai official JSON · release v2026.08.08, 7-run avg, Opus 5 low→max, score only ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
OSWorld 2.1 (O5.5)
anthropic
Opus 5.5 card p207 + Sonnet 5.5 card p131 · v2.1 files of 10 Sep, partial credit, 3 models × 5 efforts (effort from cost order) + Sonnets at max ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max · Sonnet 5·max
OSWorld 2.1 strict
anthropic
Sonnet 5.5 card p132 · strict pass, Opus 5.5 × 5 (effort from cost order) + Sonnets at max; other Sonnet points held ✓sweepSonnet 5.5·max · Sonnet 5·max
OSWorld eff. (4.7)
anthropic
Opus 4.7 card p209 · pass@1 vs output tokens, 3 models × low→max ✓sweepOpus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Sonnet 4.6·low · Sonnet 4.6·high
OSWorld sweep
anthropic
Opus 4.8 card p222 · sweep low→max, output tokens ✓sweepOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max
oubakiou godot-llm-gamebench: Opus 5.5 child, 5 efforts, parent fixed opus-5.5@medium
zenn-oubakiou
Claude Code CLI child at claude-opus-5-5, parent claude-opus-5-5@medium fixed; Godot 4.4.1 'Ember Warden' mini-game vs private spec, scored by 57 hidden tests; n=3 reps per effort; zenn.dev/oubakiou/articles/fa3b3914e081c8; read 2026-10-05 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
pelican-bicycle-sigma.vercel.app — Haiku 5.5 effort sweep (SVG pelican-on-bicycle)
pelican-bicycle-sigma
independent site, one Claude model (Haiku 5.5) at five efforts = five couples per RULES.md; cost is the page's own $ figure computed from a usage panel (not a billed invoice) -> list-price-cost; page's own caveat: "Usage numbers are transcribed from each run's usage panel" (manual transcription, not raw API JSON); read 2026-10-08 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
PetriBench
arxiv-2609.19883
arXiv 2609.19883 · 4,800 Petri-net questions, raw API adaptive thinking + output_config.effort, Opus 5 M/H/XH + Sonnet 5 M/H, measured tokens × list ✓sweepOpus 5·xhigh · Opus 5·high · Opus 5·medium · Sonnet 5·high · Sonnet 5·medium · Haiku 4.5·nothink
PhysicianBench
anthropic
Sonnet 5.5 card p138 · Sonnet 5.5 low→max labelled + 4 models at max, score-only (time axis) ✓sweepOpus 5·max · Sonnet 5·max
Playcode MacBook SVG
playcode
Playcode · one-shot MacBook SVG, high/xhigh/max per model, provider-billed cost (quality ordinal, not scored) ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Sonnet 5·high · Opus 4.8·high · Opus 4.8·xhigh · Sonnet 4.6·high · Sonnet 5·xhigh · Opus 4.8·max · Fable 5·high · Opus 5.5·high · Opus 5·high · Fable 5·xhigh · Fable 5.1·high · Opus 5·xhigh · Sonnet 5·max · Opus 5.5·xhigh · Opus 5·max · Fable 5·max · Opus 5.5·max · Fable 5.1·xhigh · Fable 5.1·max
Qiita yunany90 Opus 5.5 effort sweep, brick-breaker single index.html
qiita-yunany90
Claude Code (Windows; Max 20x); same request 5 times, only effort changed; cost = ccusage API-equivalent; read 2026-10-15 ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
QuoteBench
arxiv-2608.13547
arXiv 2608.13547 · 56 frozen tasks, provider-reported output tokens, 3 models × ladder ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 4.6·low · Sonnet 4.6·high
Ramen Bench
github-ramenbench
not-stbenjam/ramen-bench · one-shot creative-coding HTML, Sonnet 5.5, Opus 5.5, Opus 5, Fable 5.1 × low→max, Claude Code run.json, cost only ✓sweepFable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·xhigh
Reddit 1x0iu5x (dergachoff): Haiku 5.5 research sub-agent at three efforts vs DeepSeek
reddit-dergachoff
4 real research briefs, Haiku 5.5 low/medium/high vs DeepSeek V4.1 Flash (low), Opus blind A/B judge both orders; score = briefs won vs DeepSeek of 4 (ties counted as not won); cost per run as self-reported (scope unstated); read 2026-10-15 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high
Reddit: pelican-on-a-bicycle SVG, Opus 5.5 effort sweep
reddit-1wrdd57
one-line prompt 'Write HTML code that displays a 2D animation of a pelican riding a bicycle using SVG graphics...', fresh session each, author's own agent setup (pi with custom skills); n=1 per level, times/costs from session logs, no numeric quality score (qualitative only); read 2026-10-05 ✓sweepOpus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh
renchris review f51
github-renchris-infra
claude -p no tools, CC 2.1.260 (64K cap) · same 9-brief corpus, Fable 5.1 low→xhigh (max incomplete), 3 blind judges ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh
renchris review o55
github-renchris-infra
claude -p no tools, CC 2.1.280 · 9 code briefs / 36 anchored defects, Opus 5.5 low→max + Opus 5 @high, 3 blind judges; CLI list cost ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·high
renkon40 · 12 editable PowerPoint slides in a given style
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, 12 editable PowerPoint slides in a given style; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepSonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·xhigh
renkon40 · 3-minute talk slides
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, 3-minute talk slides; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepFable 5.1·max · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
renkon40 · dashboard from an AI-secretary log
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, dashboard from an AI-secretary log; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepOpus 5.5·high · Fable 5.1·high · Sonnet 5.5·high · Sonnet 5.5·max
renkon40 · Note article from an experiment log
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, Note article from an experiment log; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepOpus 5.5·high · Fable 5.1·high · Sonnet 5.5·high · Sonnet 5.5·max
renkon40 · short video
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, short video (60 s from a 25-min YouTube talk); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepOpus 5.5·max · Fable 5.1·max · Sonnet 5.5·max
renkon40 · YouTube-channel homepage
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, YouTube-channel homepage (no skill); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepFable 5.1·high · Opus 5.5·high · Sonnet 5.5·high · Sonnet 5.5·max
retort exp-49
github-retort
Claude Code · one task, 2 models × low→max, n=3; repo states its own quality saturation → cost only ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max
retort exp-55 brazil go
github-retort
Claude Code · brazil-soccer-mcp Go, Opus 5 low/high/xhigh/max (medium has no cost), n=1 ✓sweepOpus 5·low · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low
retort exp-55 brazil py
github-retort
Claude Code · brazil-soccer-mcp Python, Opus 5 low→max, n=1 ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low
retort exp-55 go
github-retort
Claude Code · rest-api-crud Go, Opus 5 low→max, n=2 (max n=1); coverage 1.0 → cost only ✓sweepOpus 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh
retort exp-55 py
github-retort
Claude Code · rest-api-crud Python, Opus 5 low→max, n=2 (max n=2); coverage 1.0 → cost only ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
retort exp-74 go
github-retort
Claude Code · rest-api-crud in Go, Opus 5.5 × low→max, n=3; requirement coverage 1.0 everywhere → cost only ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
retort exp-74 py
github-retort
Claude Code · rest-api-crud in Python, Opus 5.5 × low→max, n=3; requirement coverage 1.0 everywhere → cost only ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
retort exp-79 go isolated
github-retort
Claude Code (isolated, MCP disabled) · rest-api-crud in go, Opus 5.5 × low/medium/high/default, n=3; requirement coverage 1.0 everywhere → cost only; read 2026-10-05 ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
retort exp-79 python isolated
github-retort
Claude Code (isolated, MCP disabled) · rest-api-crud in python, Opus 5.5 × low/medium/high/default, n=3; requirement coverage 1.0 everywhere → cost only; read 2026-10-05 ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
RHI (Sonnet 4.6)
arxiv-2607.15524
arXiv 2607.15524 Fig. 5b · 30 ML tasks, cost normalized to sonnet-4.6-high ✓sweepSonnet 4.6·high
Self-conditioning test: 100 planted turns, 100% wrong totals (muninn.austegard)
muninn-austegard
ICLR-2026 self-conditioning re-run, 50-word dictionary, claude -p --resume, 20 sessions per cell, Haiku 5.5; read 2026-10-18 ✓sweepHaiku 5.5·high · Haiku 5.5·low
serotoninmax Haiku 5.5 effort sweep, tagging 146 Japanese book blurbs x 91 questions
zenn-serotoninmax
API claude-haiku-5-5 max_tokens 32000, output_config.effort; cached 6.4K-token system prompt; per-item cost = measured tokens x list price; Jev reference not Claude; read 2026-10-15 ✓sweepHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max
Simon Willison: pelican-riding-a-bicycle SVG test, own measured cost
simonwillison
llm CLI (llm-anthropic plugin), own measured spend (not list-price computation), n=1 per point, quality is ordinal/qualitative (no numeric score) -> cost-only; all 5 efforts same session per h4's gist; read 2026-10-08 ✓sweepHaiku 5.5·low · Haiku 5.5·max
Sonar Java
sonarsource
Sonar leaderboard JSON · 4,444 Java tasks, single-shot, measured tokens × list price; Opus 5.5 medium/high ✓sweepOpus 4.7·high · Opus 4.8·high · Opus 5·high · Opus 5.5·high · Opus 5.5·medium · Sonnet 4.6·high · Sonnet 5.5·high
SWE Refactor
arxiv-2608.23564
arXiv 2608.23564 · 20 whole-repo migrations, Claude Code, measured API spend, 2 models × ladder ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
SWE-bench Pro subset (Anthropic docs)
anthropic
Anthropic-internal 478-problem subset, customer-billed cost ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Fable 5.1·low · Fable 5.1·high
SWE-Effort 25
arxiv-2608.01347
arXiv 2608.01347 App. N · 25 hard SWE-bench Verified, Claude Code 2.1.220, Opus 5 & Sonnet 5 × low/high/xhigh, billed cost ✓sweepOpus 5·low · Opus 5·high · Opus 5·xhigh · Sonnet 5·low · Sonnet 5·high · Sonnet 5·xhigh
SWE-Pro (Fable)
anthropic
Fable 5 card p255 · Fable=Mythos 5, sweep low→xhigh, $ cost ✓sweepOpus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh
SWE-Pro sweep
anthropic
Opus 4.8 card p196 · sweep low→max, output tokens ✓sweepOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max
Synthorai (ja mirror): 13 single-shot answer tasks, Sonnet 5.5 / Sonnet 5 / Opus 5.5 by effort
synthorai
native Messages API via gateway, 13 tasks x3 reps x6 efforts, cost computed from list price; qiita.com/Synthorai/items/98e776a66f483f3ceeaf (mirror of synthorai.io/ja/blog/claude-sonnet-5-5-cost/); read 2026-10-05 ✓sweepSonnet 5.5·high · Sonnet 5·high · Sonnet 5.5·low · Sonnet 5·low · Opus 5.5·low · Sonnet 5.5·medium · Sonnet 5·medium · Opus 5.5·medium · Opus 5.5·high · Sonnet 5.5·xhigh · Sonnet 5·xhigh · Opus 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·max · Opus 5.5·max
Synthorai multi-hop tool loop (4 Qs ×12)
synthorai
Opus 5.5 default/low/max vs Opus 5 default ✓sweepOpus 5.5·low · Opus 5.5·max · Opus 5·high
Synthorai single-shot arithmetic (13 tasks ×3)
synthorai
Opus 5.5 and Opus 5 full ladders + default, API list-price cost; accuracy ≥97 % everywhere ✓sweepOpus 5·high · Opus 5.5·low · Opus 5·low · Opus 5.5·medium · Opus 5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5·xhigh · Opus 5.5·max · Opus 5·max
Tafla benchmark 2026-10-08: everyday calculations, default vs low effort
tafla.is
12 questions x 5 runs x (question + follow-up), live agent with GRID calculation engine; score = correct boards of 120 turns; cost per run (60 runs per lane); read 2026-10-15 ✓sweepSonnet 5.5·high · Haiku 5.5·low · Sonnet 5.5·low
tank_ai wareki converter
note-tank_ai
note.com 29 Sep · Claude Code subagents at fixed model × effort, n=1, 38 auto-graded tests, score only ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high
TB 3.0 (claude.dev chart)
anthropic
claude.dev 'Spending your effort' SVG · 70 tasks, Opus 5.5 / Fable 5.1 / Opus 5 / Fable 5 × low→max, median tokens, no USD; Opus 5.5 run 3 wk later, 128k cap ✓sweepFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
TB 3.0 (claude.dev payload)
anthropic
claude.dev 'Spending your effort' embedded data · 74 tasks×5, Opus 5 / Fable 5 / Fable 5.1 labelled rungs, median tokens, no USD ✓sweepOpus 5·low · Opus 5·medium · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·xhigh · Fable 5.1·max
TB 4.0 (Anthropic launch charts)
anthropic
Anthropic launch page chart CSVs (Opus 5.5 and Sonnet 5.5 pages) · 5 models × low→max; production safeguards on (fallbacks); Sonnet 5 anomalously low, flagged ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
Towards AI internal writing-Elo board, Sep 25, 2026
reddit-1wr03jc-img2
10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; Opus 5.5 six-effort scatter + Fable 5.1 reference point, footer 'Sep 25, 2026'; values are printed point labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓sweepOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·max
uhyo React proficiency
zenn-uhyo
zenn.dev uhyo, round 17 (25 Sep) · 13 React specs × 3 samples, judge pinned to Sonnet 4.6, effort pinned; score only ✓sweepOpus 5.5·max · Opus 5.5·high · Opus 5·max · Opus 5·high · Fable 5·max · Fable 5.1·high · Opus 4.8·max · Fable 5·high · Sonnet 5.5·high · Sonnet 5.5·max · Haiku 5.5·high · Haiku 5.5·max
Vals blog: Vibe Code Bench, Opus 5.5 single agent
vals.ai
Claude Code CLI headless, Opus 5.5, single agent, 50 apps, 1 run per setup; score = share of UI tests passed; cost = list-price totals reported by Claude Code per app; read 2026-10-15 ✓sweepOpus 5.5·medium · Opus 5.5·max
Vals blog: Vibe Code Bench, Opus 5.5 team (lead + up to 5 subagents)
vals.ai
Claude Code CLI headless, Opus 5.5, team, 50 apps, 1 run per setup; score = share of UI tests passed; cost = list-price totals reported by Claude Code per app; read 2026-10-15 ✓sweepOpus 5.5·medium · Opus 5.5·max
Vals VibeCode
vals.ai
Vals AI · Opus 5 sweep low→max (scores), $ cost at leaderboard setting; board entries — snapshot 27 Sep, re-priced ✓sweepOpus 5·max · Sonnet 5·max · Fable 5·max · Fable 5.2·high
vibe-openscad
github-vibeopenscad
output_config.effort explicit · 3 models × ladder; quality saturates → cost only ✓sweepFable 5.1·low · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·low · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 4.7·low · Opus 4.8·low · Sonnet 4.6·low · Sonnet 5·low · Opus 4.7·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·nothink · Opus 4.8·nothink · Sonnet 4.6·nothink · Opus 4.7·xhigh · Opus 4.8·xhigh · Sonnet 5·xhigh · Haiku 4.5·nothink · Sonnet 5·high
vlm-exam
github-vlmexam
roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓sweepFable 5·low · Fable 5·high · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Sonnet 5·high · Opus 4.8·low · Opus 4.8·high · Opus 5·low · Opus 5·high · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high
VulcanBench CII v4
github-vulcanbench
Claude Code 2.1.280 · 23 legacy binary-parity tasks, Opus 5.5 × low→max, CLI-reported cost; refusal fallback on (0→48 % of replies by Opus 4.8) ✓sweepOpus 5.5·low · Sonnet 5.5·low · Opus 5.5·medium · Sonnet 5.5·medium · Sonnet 5.5·high · Opus 5.5·xhigh · Sonnet 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·high
VulcanBench CII-v4 code_quality (LLM judge) for the held Opus 5.5 ladder
github-vulcanbench
github.com/morganlinton/VulcanBench, src/vulcan_gpt61sol-opus55-astra-quality-efforts.csv (commit 3f1bfa61, 2 Oct 2026), same combined-v3.15 Opus 5.5 ladder as group vulcanbench-ciiv4; read 2026-10-05 ✓sweepOpus 5.5·low · Sonnet 5.5·low · Opus 5.5·medium · Sonnet 5.5·medium · Sonnet 5.5·high · Opus 5.5·xhigh · Sonnet 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·high
VulcanBench F-v4 f51
github-vulcanbench
Claude Code 2.1.259–261 · same 23 tasks, Fable 5.1 × low→max, v3.4 judge, per-model list ledger; fallback 7–40 % ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max
VulcanBench v3 Opus 5
github-vulcanbench
VulcanBench harness · 23 tasks ×1, Opus 5 low/medium/high, report No.10 (26 Jul) ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high
WANDR
anthropic
Opus 5.5 card p188 · Perplexity wide-search, offline index, 980k budget, 3 models × low→max, scores printed ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max
WeirdML
epoch-archive
Epoch archive · 3 models × high/max, cost per run ✓sweepFable 5.1·max · Fable 5.1·high · Fable 5·max · Opus 5·max · Opus 5·high · Fable 5·high · Opus 4.8·xhigh · Opus 4.7·high · Opus 4.8·medium · Opus 4.7·max · Sonnet 5·high
WideSearch (Anthropic docs)
anthropic
Fable 5 low/medium/default, 3 runs ✓sweepFable 5·low · Fable 5·medium · Fable 5·high
Willison SVG 5.1
simonwillison
llm CLI · one fixed SVG prompt, Fable 5.1 low→max, measured tokens; trivial task ✓sweepFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max
wook3024 CC bench
github-wook3024
Claude Code · 3 tasks × 2 runs, Sonnet 4.6 & Opus 4.6 × medium/high (+ fixed-thinking Opus), CLI cost ✓sweepSonnet 4.6·high
Zapier AutomationB.
benchmarklist
AutomationBench 1.0.6 · 657 held-out tasks, strict pass/fail, measured tokens × list ✓sweepOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·low · Sonnet 4.6·high · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Fable 5.2·max · Fable 5.2·xhigh · Fable 5.2·high · Fable 5.2·low · Fable 5.2·medium
zenn QCD #27
zenn-nnakapa
Claude Code 2.1.197 · 3 DB tasks × 10 runs, Opus 4.8 & Sonnet 5 × low→xhigh, CLI-reconciled cost ✓sweepOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
deepseek-v41-techreport: deepswe
deepseek
DeepSWE v1.1 (Resolved), mini-SWE harness; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness mini-SWE; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓sweep+xmodelOpus 5·max
deepseek-v41-techreport: gpqa
deepseek
GPQA Diamond (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓sweep+xmodelOpus 5·max
deepseek-v41-techreport: tb21
deepseek
Terminal-Bench 2.1 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓sweep+xmodelOpus 5·max
Deictic ambiguity ablation experiment balanced accuracy
arxiv-2609.12162
Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); ablation experiment; read 2026-10-19 ✓xgenHaiku 4.5·nothink
Deictic ambiguity primary experiment balanced accuracy
arxiv-2609.12162
Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); primary experiment; read 2026-10-19 ✓xgenHaiku 4.5·nothink
FeatherBench
ed-is-ai/featherbench
OpenRouter(Anthropic) · 28-task multi-domain suite (coding/data/realworld/security/tool-use), effort=high (Haiku=solo), mean cost/tokens per task-run, consolidated results/summary.json; only Opus 5.5 rung saturated ✓xgenFable 5·high · Fable 5.1·high · Sonnet 4.6·high · Sonnet 5·high · Haiku 4.5·nothink · Opus 5·high · Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high
George Liu · Opus 4.8 vs Fable 5
ai.georgeliu
Claude Code 10-prompt suite · medium/high baselines ✓xgenOpus 4.8·medium · Opus 4.8·high · Fable 5·medium · Fable 5·high
George Liu · Sonnet 5 vs 4.6
ai.georgeliu
Claude Code 10-prompt suite · medium/high baselines ✓xgenSonnet 5·medium · Sonnet 5·high
ianlpaterson
ianlpaterson
OpenRouter · reasoning OFF ✓xgenSonnet 4.6·nothink · Haiku 4.5·nothink
MobileCybench token/cost accounting
arxiv-2609.23980
Claude Code 100-run aggregate · Opus4.8/Opus5 default, tokens+cost ✓xgenOpus 4.8·max · Opus 5·max
Official tau2-bench leaderboard, banking_knowledge (AllTools), Claude submissions by Sierra
github-sierra-tau2-bench
banking_knowledge domain, 4 trials, user simulator gpt-5.2 (low), reasoning_effort max for all; pass^1 %; read 2026-10-18 ✓xgenOpus 5·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max
ponytail
github-ponytail
Claude Code headless · thinking NS ✓xgenHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 4.8·nothink
Qiita suwa_nobu 12 models
qiita-suwanobu
Claude Code 2.1.283 · trivial read/count/write task, cold cache, median of 3, CLI cost, CLI default effort ✓xgenSonnet 4.6·high · Opus 4.7·xhigh · Opus 4.8·high · Fable 5·high · Sonnet 5·high · Opus 5·high · Fable 5.1·high · Opus 5.5·medium
retort exp-46/75 bookshop Clojure
github-retort
Claude Code · bookshop/rest-api-crud in Clojure, Opus 5 default (n=1) vs Opus 5.5 low (n=3) ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 bookshop Rust
github-retort
Claude Code · bookshop/rest-api-crud in Rust, Opus 5 default (n=1) vs Opus 5.5 low (n=3) ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil C
github-retort
Claude Code · brazil-soccer-mcp in C, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil C#
github-retort
Claude Code · brazil-soccer-mcp in C#, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil C++
github-retort
Claude Code · brazil-soccer-mcp in C++, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil Clojure
github-retort
Claude Code · brazil-soccer-mcp in Clojure, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil Elixir
github-retort
Claude Code · brazil-soccer-mcp in Elixir, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil Erlang
github-retort
Claude Code · brazil-soccer-mcp in Erlang, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil Java
github-retort
Claude Code · brazil-soccer-mcp in Java, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil Obj-C
github-retort
Claude Code · brazil-soccer-mcp in Objective-C, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil Rust
github-retort
Claude Code · brazil-soccer-mcp in Rust, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-46/75 brazil Swift
github-retort
Claude Code · brazil-soccer-mcp in Swift, Opus 5 default vs Opus 5.5 low, n=1 ✓xgenOpus 5·high · Opus 5.5·low
retort exp-79 vs exp-80 python (Opus 5.5 vs Fable 5.1, isolated)
github-retort
Claude Code (isolated) · rest-api-crud in python, prompt=neutral, effort=low, Opus 5.5 vs Fable 5.1, n=3 each side; requirement coverage 1.0 → cost only; read 2026-10-05 ✓xgenOpus 5.5·low · Fable 5.1·low
retort exp-79 vs exp-80 rust (Opus 5.5 vs Fable 5.1, isolated)
github-retort
Claude Code (isolated) · rest-api-crud in rust, prompt=neutral, effort=low, Opus 5.5 vs Fable 5.1, n=3 each side; requirement coverage 1.0 → cost only; read 2026-10-05 ✓xgenOpus 5.5·low · Fable 5.1·low
SlopCodeBench Oct 2026 (humanlayer writeup): strict checkpoints passed of 196
humanlayer-scb-2026-10
36 problems / 196 checkpoints, one campaign per model+effort; cost = recorded estimate per campaign (USD); read 2026-10-18 ✓xgenOpus 5.5·medium · Opus 5.5·xhigh · Fable 5.1·xhigh
swe-rebench
swe-rebench
ReAct minimal · Opus4.8 xhigh/4.7 high, Sonnet default ✓xgenOpus 4.8·xhigh · Opus 4.7·high
Writing-Elo effort/cost chart, Reddit 1wrkwp0 (r/LLMDevs)
reddit-1wrkwp0
same 'Writing Elo' metric family, 10 real YouTube scripts per OP, scored blind by three judges and a rubric; cost per script, log-x scatter with effort-labelled points; no board date printed on the chart; a different Reddit post/source snapshot than 1wr03jc, kept as its own board version; read 2026-10-05 ✓xgenOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·max
1M-context retrieval: multi-hop chain retrieval (15 cells: 5 chains x 256K/512K/1M)
arxiv-2605.02173
Retrieval and multi-hop reasoning in 1M-token contexts (arXiv 2605.02173, 4 May 2026) Tables 5/6: multi-hop chain retrieval (15 cells: 5 chains x 256K/512K/1M); five models via official APIs; temperature 0 where allowed (Claude Opus 4.7 and GPT-5.5 enforce 1); score = correct cells / total as %; read 2026-10-20 ✓xmodelOpus 4.7·max
34bench, single Thai bus-route scan transcription, 5 runs/model (dtinth/34bench)
dtinth/34bench
results.svg (table-in-SVG, parsed as text) for median accuracy (1-weighted-CER over header/forward/return/footer); meta/-.json cost field (OpenRouter USD, independently re-derived here as the median of the 5 runs' own `cost` fields) for cost; effort: meta.json has "effort": null and README says each model runs at its default reasoning effort, routed through OpenRouter; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high · Sonnet 5.5·high · Haiku 5.5·nothink
4DCodeBench: inverse-graphics-of-dynamic-scenes leaderboard (200 scenes), Claude Code agent
4dcodebench
arxiv.org/html/2610.03715v1 Figure 4 (src/main_leaderboard.png); Overall = mean of 5 family scores [0,1]; values printed on heatmap cells, read directly (not pixel-digitized); n=1 run/model, no repeats stated; read-from-image; read 2026-10-05 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high
AA AutomationB.
artificialanalysis
AA evaluation page · 657 tasks partial-credit, 3 models at max ✓xmodelOpus 5.5·xhigh · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max · Opus 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Haiku 5.5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Opus 4.8·max · Opus 5·high · Opus 5·low · Opus 5·medium · Opus 5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh
AA Briefcase v1.1 (xAI Grok 4.7 post)
xai
AA Briefcase v1.1 Elo, multi-hour office work; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓xmodelFable 5.1·max
AA FrontierCode 1.1 (AA-run, weighted score)
artificialanalysis
Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·max
AA MMMU-Pro
artificialanalysis
Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·xhigh · Opus 5·low · Sonnet 5·max · Opus 5·high · Opus 5·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 5·medium
AA-AnalystAgent
artificialanalysis
Artificial Analysis AA-AnalystAgent; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5·max · Opus 4.8·max · Opus 5·max · Fable 5·max · Opus 4.7·max
AA-Briefcase (xAI Grok 4.6 post)
xai
AA-Briefcase Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓xmodelFable 5·max
AA-Briefcase Elo, Artificial Analysis article of 21 July 2026
artificialanalysis-article
AA-Briefcase Elo as printed in the AA article "Kimi K3: second only to Fable 5 on AA-Briefcase" (2026-07-21); earlier snapshot than the live board; read 2026-10-20 ✓xmodelSonnet 5·max · Opus 4.8·max
aa-index411
tabbit.ai
—xmodelOpus 5·max
aa-index5
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·low · Fable 5.1·xhigh · Fable 5.1·high · Opus 5.5·xhigh · Opus 5.5·medium · Opus 5.5·high · Fable 5.2·max · Fable 5.2·xhigh · Fable 5.2·high · Fable 5.2·medium · Fable 5.2·low
aa5-aa-briefcase
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Haiku 5.5·low · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·xhigh · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Haiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Haiku 5.5·xhigh
aa5-code2video-aa
artificialanalysis
—xmodelOpus 5.5·max · Haiku 5.5·max · Sonnet 5.5·max · Fable 5.1·max
aa5-critpt-v2-hard
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·xhigh · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Haiku 5.5·xhigh
aa5-frontiercode-2-0
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·xhigh · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Haiku 5.5·xhigh
aa5-harvey-lab-aa
artificialanalysis
—xmodelHaiku 5.5·high · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Sonnet 5.5·max · Fable 5.1·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max
aa5-healthbench
artificialanalysis
—xmodelOpus 5.5·max · Haiku 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max
aa5-healthbench-professional
artificialanalysis
—xmodelOpus 5.5·max · Haiku 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max
aa5-hle-diamond
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·xhigh · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Haiku 5.5·xhigh
aa5-mka
artificialanalysis
—xmodelFable 5.2·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max
aa5-mlcr-aa
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·low · Fable 5.1·xhigh · Fable 5.1·high · Opus 5.5·xhigh · Opus 5.5·medium · Opus 5.5·high · Fable 5.2·max
aa5-omniscience-non-hallucination
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·low · Fable 5.1·xhigh · Fable 5.1·high · Opus 5.5·xhigh · Opus 5.5·medium · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max
aa5-vials-aa
artificialanalysis
—xmodelOpus 5.5·max · Haiku 5.5·max · Sonnet 5.5·max · Fable 5.2·max
aaenterpriseops
artificialanalysis
—xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Sonnet 5·xhigh
aagpqa
artificialanalysis
—xmodelFable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Fable 5·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 4.8·max · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·max
aaifbench
artificialanalysis
—xmodelFable 5·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 4.8·max
aaime25
artificialanalysis
—xmodelHaiku 4.5·nothink
aalcb
artificialanalysis
—xmodelHaiku 4.5·nothink
aaomni-index
artificialanalysis
—xmodelSonnet 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·medium · Sonnet 5.5·max · Sonnet 5.5·low · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·max · Opus 5.5·xhigh · Opus 5.5·high · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 4.5·nothink · Opus 4.7·max · Opus 5.5·low · Opus 5.5·medium · Opus 5·max · Fable 5.1·xhigh · Fable 5.1·high · Opus 5·high
aatau-banking
artificialanalysis
—xmodelFable 5·max · Opus 4.7·max · Opus 4.8·max · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·max · Fable 5.1·low · Fable 5.1·xhigh · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·high
aatau2
artificialanalysis
—xmodelFable 5·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 4.8·max
aatb21
artificialanalysis
—xmodelFable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Fable 5·max · Opus 4.7·max · Opus 4.8·max · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·max
aatbhard
artificialanalysis
—xmodelFable 5·max · Haiku 4.5·nothink · Opus 4.7·max · Opus 4.8·max
adebench
github-ade-bench
—xmodelFable 5·low · Opus 4.8·xhigh
Agents on Guren, Stage 2 re-run, bare condition (gurenjs/agents-on-guren)
gurenjs/agents-on-guren
RESULTS-STAGE2.md at commit 2b43507 ("Stage 2 re-run with auto memory off and Sonnet 5.5", 2026-09-30), bare condition; Claude Code 2.1.284, 3 trials per cell, 9 tasks; pass is hidden-behaviour-test pass/fail per cell (27 cells); cost is median API-equivalent USD per cell from Claude Code total_cost_usd; effort: run.sh passes only --model "$MODEL", no effort flag, so default for every model; read 2026-10-01 ✓xmodelOpus 5.5·medium · Fable 5.1·high
Agents on Guren, Stage 2 re-run, shipped condition (gurenjs/agents-on-guren)
gurenjs/agents-on-guren
RESULTS-STAGE2.md at commit 2b43507 ("Stage 2 re-run with auto memory off and Sonnet 5.5", 2026-09-30), shipped condition; Claude Code 2.1.284, 3 trials per cell, 9 tasks; pass is hidden-behaviour-test pass/fail per cell (27 cells); cost is median API-equivalent USD per cell from Claude Code total_cost_usd; effort: run.sh passes only --model "$MODEL", no effort flag, so default for every model; read 2026-10-01 ✓xmodelOpus 5.5·medium · Fable 5.1·high
Agents on Rails · atomic
rubyonrails
rubyonrails.org/ai + rails/ai-evals runs · 21 Writebook tasks ×3, miniswen, list-price cost/run ✓xmodelOpus 5·high · Fable 5.1·high · Fable 5·high · Opus 4.8·high · Sonnet 5·high
Agents' Last Exam (OpenAI GPT-5.6 post chart)
openai
Agents' Last Exam, professional workflows, code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓xmodelOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max
Agents' Last Exam (OpenAI GPT-6 Astra post chart)
openai
Agents' Last Exam; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
Agents' Last Exam (OpenAI GPT-6 Sol and Luna post)
openai
Agents' Last Exam; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓xmodelFable 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
Agents' Last Exam ALE-CLI (GLM-5.3 card)
zhipu
ALE official protocol, Claude Code harness, 105 tasks, effort=max 1M context 64K output, 4h timeout; GLM-5.3 card; read 2026-10-19 ✓xmodelOpus 4.8·max · Fable 5·max
AgentTime (arXiv 2610.09944) Agents’ Last Exam under duration requests
arxiv-2610.09944
AgentTime Table 5 column Agents’ Last Exam: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) AppWorld under duration requests
arxiv-2610.09944
AgentTime Table 5 column AppWorld: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) AssistantBench under duration requests
arxiv-2610.09944
AgentTime Table 5 column AssistantBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) DeepSWE v1.1 under duration requests
arxiv-2610.09944
AgentTime Table 5 column DeepSWE v1.1: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) GPQA Diamond under duration requests
arxiv-2610.09944
AgentTime Table 5 column GPQA Diamond: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) Humanity’s Last Exam under duration requests
arxiv-2610.09944
AgentTime Table 5 column Humanity’s Last Exam: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) METR public tasks under duration requests
arxiv-2610.09944
AgentTime Table 5 column METR public tasks: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) OSWorld 2.0 under duration requests
arxiv-2610.09944
AgentTime Table 5 column OSWorld 2.0: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) PaperBench under duration requests
arxiv-2610.09944
AgentTime Table 5 column PaperBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) PPTArena under duration requests
arxiv-2610.09944
AgentTime Table 5 column PPTArena: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) ProgramBench under duration requests
arxiv-2610.09944
AgentTime Table 5 column ProgramBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) Sakana ALE-Bench under duration requests
arxiv-2610.09944
AgentTime Table 5 column Sakana ALE-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) Terminal-Bench 4.0 under duration requests
arxiv-2610.09944
AgentTime Table 5 column Terminal-Bench 4.0: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) TUA-Bench under duration requests
arxiv-2610.09944
AgentTime Table 5 column TUA-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) WildClawBench under duration requests
arxiv-2610.09944
AgentTime Table 5 column WildClawBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AgentTime (arXiv 2610.09944) YC-Bench under duration requests
arxiv-2610.09944
AgentTime Table 5 column YC-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓xmodelFable 5.1·max
AI Coding Daily LLM Coding Leaderboard (4 projects, max 70)
aicodingdaily
Laravel 20 + React-TS 20 + bug finding 10 + edge-case 20 = 70; Claude Code rows; author-scored; page 'Last update: October 17th, 2026'; read 2026-10-18 ✓xmodelOpus 5.5·high · Opus 5.5·medium · Sonnet 5.5·high · Fable 5.1·medium · Opus 5.5·low · Sonnet 5.5·medium · Haiku 5.5·high · Haiku 5.5·medium
AI for Mortals · Backrooms game
aiformortals
Claude Code /goal, high effort, Blender + web game, one build per model, cost only, 29 Sep ✓xmodelSonnet 5·high · Opus 5.5·high · Fable 5.1·high
AI for Mortals · demoscene
aiformortals
Claude Code /goal, high effort, code-only demo, one build per model, cost only, earlier round ✓xmodelSonnet 5·high · Opus 5.5·high · Fable 5.1·high
AI for Mortals · launch video
aiformortals
Claude Code /goal, high effort, one build per model, no numeric score (cost, tokens, time only), 29 Sep ✓xmodelSonnet 5·high · Opus 5.5·high · Fable 5.1·high
AI for Mortals · RPG
aiformortals
Claude Code /goal, high effort, Diablo-style RPG, one build per model (no Sonnet 5), cost only, earlier round ✓xmodelOpus 5.5·high · Fable 5.1·high
AI for Mortals · website clone
aiformortals
Claude Code /goal, high effort, prompt as file, one build per model, cost only, 29 Sep ✓xmodelSonnet 5·high · Opus 5.5·high · Fable 5.1·high
aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high)
aibenchy
aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓xmodelHaiku 5.5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5·high · Opus 5·medium · Opus 5.5·medium · Sonnet 5.5·max · Opus 5·low · Fable 5·medium · Sonnet 5.5·high · Haiku 5.5·low · Sonnet 5·medium · Haiku 5.5·max · Haiku 5.5·high · Opus 4.8·medium · Opus 4.7·medium · Haiku 5.5·medium · Opus 4.8·low · Sonnet 5.5·low · Sonnet 5.5·medium
AIFM: Adizero shoe in Blender (Opus 5.5 vs Fable 5.1)
aiformortals
AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high
AIFM: Awwwards clone (Opus 5.5 vs Fable 5.1)
aiformortals
AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high
AIFM: Launch film (Opus 5.5 vs Fable 5.1)
aiformortals
AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high
AIFM: Mana Yerba Mate site (Opus 5.5 vs Fable 5.1)
aiformortals
AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high
AIFM: Moby Dick illustration in code (Opus 5.5 vs Fable 5.1)
aiformortals
AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high
AIFM: RollerCoaster Tycoon clone (Opus 5.5 vs Fable 5.1)
aiformortals
AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high
AIFM: THPS-style 3D game (Opus 5.5 vs Fable 5.1)
aiformortals
AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓xmodelOpus 5.5·high · Fable 5.1·high
aimecontract
arxiv-2608.16956
—xmodelSonnet 5·high · Fable 5·high
AIMultiple text-to-SQL (Claude Code)
aimultiple.com
AIMultiple AIM-Text-to-SQL, Opus 5.5 and Fable 5.1 in Claude Code at medium effort · SQL accuracy in %; each model answered its own subset of questions (506 and 491); no cost; read 2026-10-01 ✓xmodelOpus 5.5·medium · Fable 5.1·medium
AIReiter
aireiter
Claude Code · high ✓xmodelSonnet 5·xhigh · Opus 4.8·medium · Sonnet 5·high · Opus 4.8·high
AI生活 (YouTube): 25 questions x 99 runs on claude.ai chat, Haiku 5.5 vs Sonnet 5.5 vs Opus 5.5
youtube-aiseikatsu
claude.ai chat (not API); new chat per question; Q1-12 x5 + Q13-25 x3 = 99 runs; same grading as 29 Sep; runs on different days (Opus 26+29 Sep; Sonnet 29 Sep; Haiku 8 Oct); read 2026-10-15 ✓xmodelHaiku 5.5·high · Haiku 5.5·max · Sonnet 5.5·high · Opus 5.5·high
ALE-Bench
epoch-archive
Epoch archive · 3 models at matched high, measured token splits ✓xmodelOpus 5.5·high · Opus 5·high · Fable 5.1·high · Fable 5·high · Sonnet 5.5·high · Opus 4.8·high · Sonnet 5·high
AlgoWorlds (240 algorithmic worlds), exact optimality
arxiv-2608.29397
AlgoWorlds (arXiv 2608.29397, 29 Aug 2026): 240 algorithmic worlds, 3 trials, agent acquires information with paid tools then decides; score = exact optimality % (first final-decision metric of Table 2; feasibility, reference utility, sufficiency, coverage are other columns of the same runs); acquisition spend in tool-budget units, not USD (no cost row); read 2026-10-20 ✓xmodelOpus 4.8·max · Sonnet 5·max
AminBlg/SimpleEnglish: ASD-STE100 skill vs baseline, 9 Claude models x 8 tasks, violations per 100 words (regex linter), --effort low
github-simpleenglish
evals/results/RESULTS.md (pushed 5 Oct); score = reduction of STE violations per 100 words with the skill vs baseline (higher is better); output tokens = skill condition; read 2026-10-18 ✓xmodelFable 5.1·low · Fable 5·low · Opus 5·low · Opus 4.8·low · Sonnet 5·low
Analog Design Bench: pass@1
arxiv-2609.33356
Analog Design Bench (arXiv 2609.33356 v1 27 Sep 2026); 50 transistor-level analog design tasks x 3 rollouts = 150 two-hour attempts per configuration; main metric pass@1 (full-spec pass); GPT/Claude in native Codex/Claude Code, others via Claude Code on compatible APIs; cost = per-attempt mean; read 2026-10-20 ✓xmodelOpus 5·max · Fable 5·max
android-bench-2
google
—xmodelSonnet 5.5·max · Opus 5.5·max · Fable 5.1·high
anthropic-cookbook-latency-desk-agent
anthropic
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low
anthropic-cookbook-latency-desk-agent-fast-mode
anthropic
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5.5·low
anthropic-cookbook-latency-desk-agent-race
anthropic
—xmodelSonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high · Haiku 5.5·medium
anthropic-cookbook-latency-desk-agent-race-bigger-pages
anthropic
—xmodelSonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high · Haiku 5.5·medium
anthropic-cookbook-latency-desk-agent-race-fast
anthropic
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5.5·low
anthropic-cookbook-latency-desk-agent-race-fast-bigger-pages
anthropic
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5.5·low
anthropic-cookbook-latency-desk-agent-race-fast-prefetch
anthropic
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5.5·low
anthropic-cookbook-latency-desk-agent-race-fast-prefetch-bigger-pages
anthropic
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5.5·low
anthropic-cookbook-latency-desk-agent-race-prefetch
anthropic
—xmodelSonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high · Haiku 5.5·medium
anthropic-cookbook-latency-desk-agent-race-prefetch-bigger-pages
anthropic
—xmodelSonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high · Haiku 5.5·medium
APEX-Agents (xAI Grok 4.6 post)
xai
APEX-Agents %, harness not stated (Mercor's benchmark by name; page does not attribute); xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓xmodelFable 5·max
APEX-Harm mean task score under attack (all 6 categories)
apexharm-2610.13623
APEX-Harm 180 attack task instances per model (15 underlying APEX-Agents tasks x 12 attack variants), APEX-Agents harness with MCP tools, 150 steps, no timer, judge/rules per paper; metric mean task score 0-1 (utility under attack); ASR/ER safety rates not extracted; arXiv 2610.13623 Table 4 (Oct 2026); cost = Table 1 total for Gemini 3.8 Flash only; read 2026-10-20 ✓xmodelOpus 5·max · Sonnet 5·max
apexharm-asr
apexharm-2610.13623
—xmodelOpus 5·max · Sonnet 5·max
ARC-AGI-3 (semi-private)
arcprize
ARC Prize ARC-AGI-3 semi-private interactive-game set, score in % (as published x100), cost = total run USD; Haiku 5.5 on the Standard harness (the Provider-Adapter harness entry, 2.82 % / $437.14, is a different configuration and is not mixed in); read 2026-10-15 ✓xmodelOpus 4.7·high · Opus 4.8·high · Opus 5·high · Haiku 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Fable 5.2·xhigh · Fable 5.2·high · Fable 5.2·medium · Fable 5.2·low
arcagi3-provider-adapter
arcprize
—xmodelFable 5.2·max · Fable 5.2·xhigh · Fable 5.2·high · Fable 5.2·medium · Fable 5.2·low
Argo-Bench: enterprise data-agent tasks (210 tasks), Inspect AI 0.3.263 harness
argo-bench
arxiv.org/html/2610.02122v1 Table 2 (best-effort snapshot, extra-high where offered) + Table 10 (effort ladder, mean score /100, tasks completed at every level); n=1 rollout/task; score = mean task score out of 100 with a model's forecast tasks floored at 0 as a block; cost = API spend per task, excluding BigQuery (Table 2 only, not itemized by effort in Table 10); read 2026-10-05 ✓xmodelOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
ArtifactArena model leaderboard, Design Lab Harness (DLH) Elo
arxiv-2610.06511
Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓xmodelFable 5.1·high · Fable 5·high · Sonnet 5·high · Opus 4.8·high
ArtifactArena model leaderboard, Sampling Harness (SH) Elo
arxiv-2610.06511
Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓xmodelFable 5.1·high · Fable 5·high · Sonnet 5·high · Opus 4.8·high
ArtifactArena model leaderboard, Verifier-Grounded Refinement Harness (VGH) Elo
arxiv-2610.06511
Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓xmodelFable 5.1·high · Fable 5·high · Sonnet 5·high · Opus 4.8·high
Artificial Analysis Coding Agent Index v1.1 (OpenAI GPT-5.6 post chart)
openai
Artificial Analysis Coding Agent Index v1.1; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·medium · Opus 4.8·max
Artificial Analysis Coding Agent Index v1.4 (OpenAI GPT-6 Astra post chart, non-code)
openai
Artificial Analysis Coding Agent Index v1.4; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
Artificial Analysis Intelligence Index (OpenAI GPT-5.5 post chart)
openai
Artificial Analysis Intelligence Index; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓xmodelOpus 4.7·max
arxiv-2609.33055-weighted-sd
arxiv-2609.33055
—xmodelSonnet 5·high
arxiv-2610.02204-capagent0-mean-success
arxiv-2610.02204
—xmodelOpus 5·xhigh
arxiv-2610.04976-pricedguidance-dev-fable51-median-bits
arxiv-2610.04976
—xmodelFable 5.1·high · Fable 5.1·max
arxiv-2610.04976-pricedguidance-dev-opus5-median-bits
arxiv-2610.04976
—xmodelOpus 5·high · Opus 5·max
arxiv-2610.04976-pricedguidance-test-median-bits
arxiv-2610.04976
—xmodelFable 5.1·max · Opus 5·high
arxiv-2610.05348-hotpotqa-abst
arxiv-2610.05348
—xmodelSonnet 5.5·high
arxiv-2610.05348-nq-abst
arxiv-2610.05348
—xmodelSonnet 5.5·high
arxiv-2610.06668-tier2reject
arxiv-2610.06668
—xmodelFable 5.1·high · Sonnet 5.5·high · Opus 5·high · Sonnet 5·high · Opus 4.8·high
arxiv-2610.07563-householdbench-avg-rank
arxiv-2610.07563
—xmodelFable 5.1·high · Opus 5·high · Opus 4.8·nothink
arxiv-2610.12596-spirebench-acts
arxiv-2610.12596
—xmodelFable 5.1·low · Fable 5.1·max · Opus 5·low · Opus 5·max · Sonnet 5·low · Sonnet 5·max
arxiv-2610.14397-realitybench-doc
arxiv-2610.14397
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.17016-covert-kl-nats
arxiv-2610.17016
—xmodelSonnet 5.5·medium
arxiv-2610.17016-credit-kl-nats
arxiv-2610.17016
—xmodelSonnet 5.5·medium
arxiv-2610.17016-peer-kl-nats
arxiv-2610.17016
—xmodelSonnet 5.5·medium
arxiv-2610.18449-asb-rate
arxiv-2610.18449
—xmodelOpus 5.5·max · Sonnet 5.5·high
arxiv-2610.22599-direction
arxiv-2610.22599
—xmodelOpus 5.5·max · Fable 5.1·max
arxiv-2610.22599-r2
arxiv-2610.22599
—xmodelOpus 5.5·max · Fable 5.1·max
arxiv-2610.22599-slope
arxiv-2610.22599
—xmodelOpus 5.5·max · Fable 5.1·max
arxiv-2610.22599-top-idea-gain
arxiv-2610.22599
—xmodelOpus 5.5·max · Fable 5.1·max
arxiv-2610.25920-algoveri-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-algoveri-maestro
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-axl03-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-axl03-cc-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-axl03-maestro
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-axl03-maestro-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-axl06-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-axl06-cc-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-axl06-maestro
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-axl06-maestro-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-fatex-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-fatex-cc-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-fatex-maestro
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-fatex-maestro-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-fatex-naive
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-fqb-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-fqb-cc-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-fqb-maestro-rep
arxiv-2610.25920
—xmodelFable 5·high
arxiv-2610.25920-imo26-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-lib-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-lib-maestro
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-ntp4vc-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-ntp4vc-maestro
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-putnam-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-putnam-maestro
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.25920-verina-cc
arxiv-2610.25920
—xmodelOpus 5·high · Opus 4.8·high · Fable 5·high
arxiv-2610.29207-t13-h3-d
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
arxiv-2610.29207-t3-correct
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 5.5·medium
arxiv-2610.29207-t3-outdated
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 5.5·medium
arxiv-2610.29207-t3-uninterpretable
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 5.5·medium
arxiv-2610.29207-t3-wrong
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 5.5·medium
arxiv-2610.29207-t4-h1-d
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 5.5·medium
arxiv-2610.29207-t4-h2-s
arxiv-2610.29207
—xmodelSonnet 4.6·nothink · Opus 5.5·medium
arxiv-2610.29207-t5-outdated-count
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 5.5·medium
arxiv-2610.29207-t8-correct
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
arxiv-2610.29207-t8-outdated
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
arxiv-2610.29207-t8-uninterpretable
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
arxiv-2610.29207-t8-wrong
arxiv-2610.29207
—xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
arxiv-2610.34435-asr-all-allrev
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-all-gpt6sol
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-all-opus48
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-all-sonnet5
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-fix-allrev
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-fix-gpt6sol
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-fix-opus48
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-fix-sonnet5
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-window-allrev
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-window-gpt6sol
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-window-opus48
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-asr-window-sonnet5
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-all-glm-cc
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-all-glm-codex
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-all-gpt6sol
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-all-grok47
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-all-kimik3
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-all-opus48
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-all-sonnet5
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-all
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-glm-cc
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-glm-codex
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-gpt6sol
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-grok47
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-kimik3
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-opus48
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-fix-sonnet5
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-task-all
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-task-glm-cc
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-task-glm-codex
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-task-sonnet5
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-all
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-glm-cc
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-glm-codex
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-gpt6sol
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-grok47
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-kimik3
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-opus48
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-catch-window-sonnet5
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.34435-reviewer-catch-rate
arxiv-2610.34435
—xmodelSonnet 5·high · Opus 4.8·high
arxiv-2610.36434-geomean12-justgoal-single
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-geomean12-kda05-repro
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-dsa-heavy-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-dsa-heavy-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-dsa-mixed-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-dsa-mixed-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-dsa-small-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-dsa-small-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-fa4-bwd-gqa-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-fa4-bwd-gqa-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-fa4-bwd-mha-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-fa4-bwd-mha-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-fa4-bwd-varlen-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-fa4-bwd-varlen-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-gdn-few-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-gdn-few-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-gdn-long-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-gdn-long-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-gdn-short-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-gdn-short-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-moe-medium-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-moe-medium-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-moe-prefill-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-moe-prefill-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-moe-token-justgoal
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t7-moe-token-kda05
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t8-dsa-fwd-justgoal-whole
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t8-fa4-bwd-justgoal-whole
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t8-gdn-fwd-justgoal-whole
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.36434-t8-moe-fp8-justgoal-whole
arxiv-2610.36434
—xmodelOpus 4.8·high · Opus 5·high · Fable 5.1·high
arxiv-2610.38096-s14-robust-beams-rejected-bt
arxiv-2610.38096
—xmodelHaiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s14-robust-beams-rejected-bu-retest
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s14-robust-beams-rejected-bx
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s14-robust-beams-rejected-c4-bu
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-c4-u-accepted
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-c4-u-flagged
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-c4-u-noans
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-c4-u-rejected
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-c4-u-solved
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-c4-u-vnoans
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-c4-u-wrong
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-s2-t02-fine-u-accepted
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s2-t02-fine-u-rejected
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s2-t02-fine-u-solved
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s2-t02-main-t-accepted
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-t-flagged
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-t-rejected
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-t-solved
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-t-wrong
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-u-accepted
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-u-flagged
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-u-noans
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-u-vnoans
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-u-wrong
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-main-x-accepted
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t02-main-x-flagged
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t02-main-x-noans
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t02-main-x-rejected
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t02-main-x-solved
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t02-main-x-vnoans
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t02-main-x-wrong
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t02-pilot-u-accepted
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-pilot-u-rejected
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s2-t02-pilot-u-solved
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-fine-u-rejected
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s2-t11-main-t-accepted
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-t-flagged
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-t-rejected
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-t-solved
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-u-accepted
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-u-flagged
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-u-noans
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-u-vnoans
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-u-wrong
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-main-x-accepted
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t11-main-x-flagged
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t11-main-x-noans
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t11-main-x-rejected
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t11-main-x-solved
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t11-main-x-vnoans
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t11-main-x-wrong
arxiv-2610.38096
—xmodelFable 5.1·high · Opus 5·high
arxiv-2610.38096-s2-t11-pilot-u-rejected
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s2-t11-pilot-u-solved
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s33-t02-feasibility-only-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t02-pa-limit-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t02-pa-note-rejected
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t02-pa-note-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t02-s1-limit-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t02-s1-note-rejected
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t02-s1-note-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t11-feasibility-only-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t11-pa-limit-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t11-pa-note-rejected
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t11-pa-note-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t11-s1-limit-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t11-s1-note-rejected
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s33-t11-s1-note-valid-solved
arxiv-2610.38096
—xmodelOpus 5·high
arxiv-2610.38096-s7-t02-fine-max-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s7-t02-fine-median-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s7-t02-main-capable
arxiv-2610.38096
—xmodelOpus 4.8·high
arxiv-2610.38096-s7-t02-main-max-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s7-t02-main-median-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s7-t02-pilot-max-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s7-t02-pilot-median-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s7-t11-fine-max-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s7-t11-fine-median-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Fable 5.1·high
arxiv-2610.38096-s7-t11-main-max-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s7-t11-main-median-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-s7-t11-pilot-max-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-s7-t11-pilot-median-error
arxiv-2610.38096
—xmodelOpus 5.5·high · Opus 4.8·high
arxiv-2610.38096-t1-beam
arxiv-2610.38096
—xmodelSonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high
arxiv-2610.38096-t1-disk
arxiv-2610.38096
—xmodelSonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high · Opus 5·high · Opus 4.8·high
arxiv-2610.38096-t1-noans
arxiv-2610.38096
—xmodelHaiku 5.5·high
arxiv-2610.38096-t1-probes
arxiv-2610.38096
—xmodelOpus 4.8·high
arxiv-2610.38096-t1-robust
arxiv-2610.38096
—xmodelHaiku 5.5·high · Opus 5·high
arxiv-2610.38096-t1-vbeam
arxiv-2610.38096
—xmodelSonnet 5.5·high · Haiku 5.5·high · Opus 4.8·high
arxiv-2610.38096-t1-vdisk
arxiv-2610.38096
—xmodelSonnet 5.5·high · Haiku 5.5·high · Opus 4.8·high
arxiv-2610.40503-arcagi3-public-standard
arxiv-2610.40503
—xmodelOpus 5·high
arxiv-2610.41076-baba-difficulty-hard
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-baba-difficulty-light
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-baba-difficulty-standard
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-baba-issue-completion
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-baba-pass1-strict
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-baba-pass3
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t13-anyseed
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t14-hard
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t14-light
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t14-std
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t15-seed1-outcome
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t15-seed1-strict
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t15-seed2-outcome
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t15-seed2-strict
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t15-seed3-outcome
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2610.41076-t15-seed3-strict
arxiv-2610.41076
—xmodelOpus 5·max
arxiv-2611.05147-gm-league-arena-baseball-cash
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-baseball-elo
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-football-cash
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-football-elo
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-hockey-cash
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-hockey-elo
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-cash
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-elo
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-gi
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-losses
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-wins
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-wins-2022
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-wins-2023
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-wins-2024
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-wins-2025
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.05147-gm-league-arena-nba-wins-2026
arxiv-2611.05147
—xmodelOpus 5.5·high · Sonnet 5.5·high · Fable 5.1·high
arxiv-2611.06135-p1-f1-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-f1-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-fn-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-fn-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-fp-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-fp-pat
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-fp-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-hit-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-mcc-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-mcc-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-tp-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p1-tp-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium
arxiv-2611.06135-p2-conf1-pat
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-conf1-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-f1-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-f1-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-fn-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-fn-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-fp-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-fp-pat
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-fp-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-hit-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-mcc-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-mcc-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-tp-all
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-p2-tp-vul
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-c-f1
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-c-mcc
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-c-pc
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-f1
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-mcc
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-pb
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-pc
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-pr
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-pv
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-rr-f1
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-rr-mcc
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06135-t5-rr-pc
arxiv-2611.06135
—xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 5.5·xhigh
arxiv-2611.06841-serverbench-dpd-gemma-e4b
arxiv-2611.06841
—xmodelOpus 5.5·max · Sonnet 5.5·max · Opus 5·max
arxiv-2611.06841-serverbench-dpd-qwen35-9b
arxiv-2611.06841
—xmodelOpus 5.5·max · Sonnet 5.5·max · Opus 5·max
arxiv-2611.06841-serverbench-dpd-qwen38-27b-int4
arxiv-2611.06841
—xmodelOpus 5.5·max · Sonnet 5.5·max · Opus 5·max
arxiv-2611.07461-t12-delta-mini
arxiv-2611.07461
—xmodelFable 5·medium · Opus 5·medium · Sonnet 5·medium · Opus 4.8·medium · Opus 4.7·medium
arxiv-2611.07461-t12-hermes-resolved
arxiv-2611.07461
—xmodelFable 5·medium · Opus 5·medium · Sonnet 5·medium · Opus 4.8·medium · Opus 4.7·medium
arxiv-2611.07461-t12-mini-resolved
arxiv-2611.07461
—xmodelFable 5·medium · Opus 5·medium · Sonnet 5·medium · Opus 4.8·medium · Opus 4.7·medium
arxiv-2611.07461-t13-baseline-resolution
arxiv-2611.07461
—xmodelOpus 5·medium · Opus 5·max · Fable 5·max · Opus 4.8·max · Sonnet 5·medium · Sonnet 5·max
arxiv-2611.07461-t13-delta
arxiv-2611.07461
—xmodelOpus 5·medium · Opus 5·max · Fable 5·max · Opus 4.8·max · Sonnet 5·medium · Sonnet 5·max
arxiv-2611.07461-t13-hermes-resolution
arxiv-2611.07461
—xmodelOpus 5·medium · Opus 5·max · Fable 5·max · Opus 4.8·max · Sonnet 5·medium · Sonnet 5·max
arxiv-2611.07461-t14-baseline-avg
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-baseline-build
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-baseline-issue
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-baseline-monitor
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-baseline-test
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-delta
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-hermes-avg
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-hermes-build
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-hermes-issue
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-hermes-monitor
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t14-hermes-test
arxiv-2611.07461
—xmodelOpus 5·medium · Sonnet 5·medium
arxiv-2611.07461-t31-delta
arxiv-2611.07461
—xmodelOpus 5·medium · Opus 5·high · Sonnet 5·medium · Sonnet 5·high
arxiv-2611.07461-t31-hermes-comp
arxiv-2611.07461
—xmodelOpus 5·medium · Opus 5·high · Sonnet 5·medium · Sonnet 5·high
arxiv-2611.07667-doubleauction-coarse
arxiv-2611.07667
—xmodelOpus 5.5·low
arxiv-2611.07667-doubleauction-full
arxiv-2611.07667
—xmodelOpus 5.5·low
arxiv-2611.07962-tcsexam-lite-constructions
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-lite-index
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-lite-open-conjectures
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-lite-solver
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-lite-theorems
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t25-corners-theorem
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t25-discrete-bidding-games
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t25-folded-rs-codes
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t25-quantum-cryptography
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t25-small-depth-negations
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t26-bin-packing
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t26-course-timetabling
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t26-machine-reassignment
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t26-maximum-cut
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t26-quantified-boolean
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t28-branching-programs
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t28-evasiveness
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t28-frege-proof-length
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t28-log-rank
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.07962-tcsexam-t28-online-matching
arxiv-2611.07962
—xmodelOpus 5.5·max
arxiv-2611.08209-roboarcade-success-multiview-rgbd
arxiv-2611.08209
—xmodelFable 5.1·xhigh
arxiv-2611.08209-roboarcade-success-via
arxiv-2611.08209
—xmodelOpus 5·xhigh · Fable 5.1·xhigh
arxiv-2611.08209-roboarcade-t6-so-cost-via
arxiv-2611.08209
—xmodelOpus 5·xhigh · Fable 5.1·xhigh
arxiv-2611.08472-xstack-pass-cheat
arxiv-2611.08472
—xmodelOpus 5·high
arxiv-2611.08472-xstack-pass-correctness
arxiv-2611.08472
—xmodelOpus 5·high
arxiv-2611.08472-xstack-pass-target
arxiv-2611.08472
—xmodelOpus 5·high
arxiv-2611.08472-xstack-pass-weights
arxiv-2611.08472
—xmodelOpus 5·high
arxiv-2611.09802-t2-location-corrected-fixed
arxiv-2611.09802
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium
arxiv-2611.09802-t2-location-removed-fixed
arxiv-2611.09802
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium
arxiv-2611.09802-t2-location-stale-fixed
arxiv-2611.09802
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium
arxiv-2611.09802-t2-procedure-corrected-fixed
arxiv-2611.09802
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium
arxiv-2611.13246-expertcrowd-swe-macro-score
arxiv-2611.13246
—xmodelOpus 5.5·max · Fable 5.1·max
arxiv-2611.16591-inference-income
arxiv-2611.16591
—xmodelOpus 5.5·medium · Opus 5.5·high
arxiv-2611.16591-inference-income-multiagent
arxiv-2611.16591
—xmodelOpus 5.5·high
arxiv-2611.16591-singularity-index
arxiv-2611.16591
—xmodelOpus 5.5·max
arxiv-2611.17955-webarenapro-sr
arxiv-2611.17955
—xmodelOpus 5.5·high
Atomic Agent expense-lib
atomicagent
Atomic Agent 0.6.5 via OpenRouter · 5 bugs + 2 features, 32 hidden tests, n=3, provider default effort, billed cost; saturated ✓xmodelOpus 5.5·high · Sonnet 5·high
atomicagent-warehouse-59-hidden-tests
atomicagent
—xmodelFable 5.2·medium · Fable 5.2·high · Opus 5.5·high
AutomationB.
arxiv-2604.18934
harness+effort NOT stated ✓xmodelOpus 4.7·max
AutomationBench (OpenAI GPT-5.6 post chart)
openai
AutomationBench pass rate, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
AutomationBench (OpenAI GPT-6 Astra post chart, non-code mode)
openai
AutomationBench; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max
Bakery website, Sonnet 5.5 vs Opus 5.5 at high (AI Field Notes video)
youtube-aifieldnotes
Claude Code; same brief; empty folder; high effort; one try; 2026-10-10; read 2026-10-18 ✓xmodelSonnet 5.5·high · Opus 5.5·high · Haiku 5.5·high
bangla-llm-bench, belebele track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelOpus 5.5·medium
bangla-llm-bench, bnexam track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelOpus 5.5·medium
bangla-llm-bench, mgsm track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=250 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelOpus 5.5·medium
bangla-llm-bench, mmlu track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=285 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelOpus 5.5·medium
Bazaar multi-attribute auction, cumulative profit
bazaar-2608.00102
Bazaar sealed-bid multi-attribute auction, ~79 rounds, 10 seeds per row (GPT-5.5 xhigh 3 seeds), closed-form evaluation, no tools beyond bidding API; metric cumulative profit USD (game money, not spend; efficiency eta of Table 6 = same runs, not kept); arXiv 2608.00102 Table 3 (Aug 2026); read 2026-10-20 ✓xmodelOpus 4.7·xhigh · Opus 4.7·medium
Binder design (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelSonnet 5·max · Fable 5.1·max
Bio-imaging (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelSonnet 5·max · Fable 5.1·max
BioMystery hard-dist (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelFable 5.1·max · Opus 5.5·max
BioMystery hard-sub (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelFable 5.1·max · Opus 5.5·max
Bito agentic coding battery
bito.ai
Bito's internal agentic coding battery in Claude Code (6 task types, multi-turn sessions), each turn scored out of 50 by Opus 5.5 · mean score /50, cost per session (USD); effort from the post; read 2026-10-01 ✓xmodelSonnet 5.5·medium · Sonnet 5.5·high
bito-s55o55-pct
bito.ai
—xmodelHaiku 5.5·medium · Haiku 5.5·high · Sonnet 5.5·high · Sonnet 5.5·medium · Opus 5.5·medium
BixBench3 mean artifact score (20 comp-bio tasks)
bixbench3-2608.25286
BixBench3 20 tasks / 138 artifacts, Inspect AI ReAct agent, 5 tools, 24 h / 5000 message cap, programmatic artifact grading, 20 runs per model (260 runs), each model at its maximum thinking effort; metric mean task score 0-1; cost = mean USD per task over 20 tasks from per-call tokens x provider list prices; arXiv 2608.25286 (Aug 2026); read 2026-10-20 ✓xmodelOpus 4.8·max · Opus 5·max
bsb-v1-green
github-bullshitbench-v1
—xmodelOpus 4.8·nothink · Sonnet 4.6·high · Sonnet 5·low · Sonnet 5·max · Opus 4.7·max · Opus 5·xhigh · Sonnet 4.6·nothink · Opus 4.8·xhigh · Haiku 4.5·nothink · Opus 4.7·nothink · Fable 5.1·low · Opus 5·low · Fable 5.1·max · Opus 5.5·low · Opus 5.5·max
bullshitbench-v1-strict
github-cc-83510
—xmodelSonnet 4.6·high · Sonnet 5·low · Opus 4.7·max · Sonnet 5·max · Opus 5·xhigh
BVB
arxiv-2609.15478
arXiv 2609.15478 · Mini-BVB via LiteLLM, $3/scene cap, 5 Claude models at high (+ omitted-effort runs), LiteLLM list cost ✓xmodelSonnet 5.5·xhigh · Opus 5.5·xhigh · Opus 5·high · Opus 5.5·high · Sonnet 5·high · Sonnet 4.6·high · Opus 4.7·high · Opus 4.8·high
c0ze/agent-evaluation: 13 tasks, blind-graded (24-way pass of 13 Oct)
c0ze-agent-evaluation
13 self-contained tasks, 1 run per model, blind rubric grading by Opus 5.5 subagents; score = mean of 13 task totals /10; cost per 13-task run (credit-balance drop, whole $; Haiku API-equivalent); read 2026-10-15 ✓xmodelSonnet 5.5·xhigh · Fable 5.1·xhigh · Opus 5.5·high · Haiku 5.5·xhigh
CalorieBench
github-caloriebench
Anthropic API · 200 food photos, effort=high sent without thinking → nothink regime, within-20 % score ✓xmodelFable 5.1·high · Opus 5.5·high · Sonnet 5·high
caloriebench-mae
github-caloriebench
—xmodelFable 5.1·high · Opus 5.5·high · Sonnet 5·high
cauchye_asia Japanese text + vision, basic set (21 q x 2)
zenn-cauchye_asia
paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓xmodelSonnet 5.5·high
cauchye_asia Japanese text + vision, hard set (10 q x 2)
zenn-cauchye_asia
paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓xmodelSonnet 5.5·high
CEO-Bench
arxiv-2606.18543
terminal-agent · Opus/Sonnet=MAX, Haiku=thinking ✓xmodelFable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 4.6·high
cfg-bash
computingforgeeks
—xmodelOpus 5·high · Opus 4.8·nothink
cfg-k8s
computingforgeeks
—xmodelOpus 5·high · Opus 4.8·nothink
cfg-terraform
computingforgeeks
—xmodelOpus 5·high · Opus 4.8·nothink
ChaosCypher context-extraction: carrier suite (N=21)
github-chaoscypher
War & Peace fixture, mcp:claude-code harness, carrier suite, pass/fail verdicts; read 2026-10-05 ✓xmodelOpus 5.5·high · Sonnet 5.5·high
ChaosCypher context-extraction: chat suite (N=80)
github-chaoscypher
War & Peace fixture, mcp:claude-code harness, chat suite, pass/fail verdicts; read 2026-10-05 ✓xmodelOpus 5.5·high · Sonnet 5.5·high · Sonnet 5·high
ChaosCypher context-extraction: native suite (N=65)
github-chaoscypher
War & Peace fixture, mcp:claude-code harness, native suite, pass/fail verdicts; read 2026-10-05 ✓xmodelOpus 5.5·high · Sonnet 5.5·high · Sonnet 5·high
Chartography (Surge AI) launch snapshot July 2026: cells absent from or different in the held board
chartography-2608.10677
Chartography launch leaderboard (arXiv 2608.10677, frozen July 2026 snapshot; 100 tasks x 20 trials, no tools, mean pass@1 % with 95% CI, Surge-hosted): only the cells that are NOT identical to the held surgehq.ai board (read 2026-10-15, group surge-chartography) are kept; join with that group for the max/xhigh peers; arXiv 2608.10677 (Aug 2026); read 2026-10-20 ✓xmodelFable 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max · Opus 4.8·nothink
Chess puzzle accuracy, base prompt (imec APO paper)
imec-2610.00416
559-puzzle test set, base prompt (no optimisation), mean of 3 runs, generative single-move answer, legal-move list given, no tools; metric puzzle accuracy %; cost = USD per 1,000 solver-position calls; arXiv 2610.00416 Table 2 (Oct 2026); read 2026-10-20 ✓xmodelHaiku 4.5·nothink
ChronoBay watch reference identification from one photo (QutaibaAbualrob/chronobay_ai_bench)
QutaibaAbualrob/chronobay_ai_bench
REPORT.md 2026-09-30, 100 images, 1 pass/image, same prompt (sha 4e521b10...); score is exact-reference-number match %; cost is per-image USD computed by the repo from its own token counts and a pinned price table (list-price-cost); read 2026-10-01 ✓xmodelOpus 5.5·medium · Haiku 4.5·nothink
Claude Code Camp 12-task
claudecodecamp
beehiiv newsletter · Claude Code, default effort, 12 tasks×3 reps, cost+tokens only ✓xmodelOpus 5.5·medium
claude-maintain-rails-evals: planner variant v12 (release notes), Claude headless, 5 runs/config
github-claude-maintain-rails-evals
claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓xmodelOpus 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high
claude-maintain-rails-evals: planner variant v16 (Argon clean-room procedure), Claude headless, 5 runs/config
github-claude-maintain-rails-evals
claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓xmodelOpus 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high
claude-maintain-rails-evals: risk-report variant v20 (changes-first), Opus 5.5 high vs low, 5 runs/config
github-claude-maintain-rails-evals
claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓xmodelOpus 5.5·high · Opus 5.5·low
claude.dev hillclimb
anthropic
claude.dev hillclimb post · 30 support tickets, audited prompt, Opus 5.5 vs Sonnet 5 at low, cost per ticket ✓xmodelOpus 5.5·low · Sonnet 5·low
ClerkBench banded_meter_billing (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload
zenodo-23048151
Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelOpus 5.5·high · Opus 5·high
ClerkBench fee_ledger_dropzero (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload
zenodo-23048151
Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelOpus 5.5·high · Opus 5·high
cliphou.se Motion Graphics Benchmark v1.1: cost per run (Remotion videos)
cliphou.se
3 briefs x 2 runs via OpenRouter, reasoning high; cost = mean OpenRouter generation cost per run over the 6 runs (computed by us); no quality score published (human review pending); read 2026-10-15 ✓xmodelOpus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high
Code-as-Policy / RoboDojo (2609.39018) — 5 simulated manipulation tasks, +URAI interface
arxiv-2609.39018
Table 1, avg success rate % over 5 tasks x 5 episodes, output tokens k incl. reasoning; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelFable 5.1·xhigh · Opus 5.5·xhigh
Code-as-Policy / RoboDojo (2609.39018) — 5 simulated manipulation tasks, native interface
arxiv-2609.39018
Table 1, avg success rate % over 5 tasks x 5 episodes, output tokens k incl. reasoning; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelFable 5.1·xhigh · Opus 5.5·xhigh
code-review-bench · /ce-code-review
github-codereviewbench
kamui/code-review-bench 23716f7 · 12 OSS PRs × 3 trials, same skill snapshot, Claude Code 2.1.284 high, recall and cohort cost ✓xmodelSonnet 5.5·high · Opus 5.5·high
code-review-bench · /thermo review
github-codereviewbench
kamui/code-review-bench 23716f7 · 12 OSS PRs × 3 trials, same skill snapshot, Claude Code 2.1.284 high, recall and cohort cost ✓xmodelSonnet 5.5·high · Opus 5.5·high
code-review-bench: Claude 5.5 as claim labeller (12 batches, 493 claims), agreement with GPT-6.1 Sol labels
github-codereviewbench
kamui/code-review-bench grader-model-test-2026-10-12; 12 sessions (12 batches of 53 reviews; 493 claims); Claude Code high; score = label agreement with the saved GPT-6.1 Sol labelling (not accuracy); read 2026-10-15 ✓xmodelOpus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high
Code4Scene public set (95 cases), overall S_model
arxiv-2609.36777
Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; public set (95 cases), overall S_model; read 2026-10-19 ✓xmodelFable 5.1·max · Opus 5·max
CodeActionBench, Reference harness (single code-generation loop)
arxiv-2609.33807
CodeActionBench (arXiv 2609.33807, 27 Sep 2026): 25 tasks x 3 seeds = 75 attempts per configuration, agentic Code-as-Policy for embodied manipulation; 'Reference' harness for all rows of this group; score = Success (/75 attempts); task coverage /25 is another summary; cost = Table 11 total API-equivalent USD over the 75 attempts (computed by the authors from tokens at list rates); read 2026-10-20 ✓xmodelOpus 5·high · Sonnet 5·high
CodeActionBench, vendor harnesses (Codex CLI / Claude Code)
arxiv-2609.33807
CodeActionBench: same 75 attempts; vendor harness rows (GPT-6 Astra in Codex CLI, Claude Opus 5 in Claude Code); Success (/75); cost = Table 11 total API-equivalent USD; read 2026-10-20 ✓xmodelOpus 5·high
CodeRabbit review (Fable 5.1)
coderabbit
CodeRabbit blog 1 Sep · 45 tasks/105 issues, Fable 5.1 low/high + Fable 5, Opus 5, score only ✓xmodelFable 5.1·low · Fable 5.1·high
Codex Radar intelligence-efficiency board, Codex CLI harness
codexradar
Codex Radar recurring repository-level coding eval (per-task pass over valid tasks; payload mode equal_latest_3: equal-weight latest three valid samples per task); harness codex; snapshot payload stamped 2026-10-01T23:00:27+08:00 read 2026-10-20; score=passed/valid_tasks (derived; payload gives passed, valid_tasks, iq=150*pass_rate); cost=average_price_usd per run (estimated USD-equivalent from token prices); valid_tasks differ per row (36..336); read 2026-10-20 ✓xmodelOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·xhigh · Opus 5.5·max
computingforgeeks: Claude Code bug-fix task (log parser)
computingforgeeks
4-bug Python log parser + 8-test pytest suite, headless Claude Code, 3 runs/model, default effort per model; score saturated (9/9 green) -> cost only; read 2026-10-01 ✓xmodelSonnet 5·high · Opus 5.5·medium
Conceptual Reasoning Index · DTBench
conceptualreasoning.ai
Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 5.5·xhigh · Opus 5.5·medium · Fable 5.1·medium · Fable 5.1·max · Opus 5·xhigh · Opus 5.5·low · Fable 5.1·xhigh · Opus 5·high · Sonnet 5.5·medium · Fable 5.1·high · Sonnet 5.5·high · Opus 5·max · Sonnet 5.5·low · Fable 5·max · Opus 5·medium · Fable 5.1·low · Fable 5·xhigh · Opus 5·low · Fable 5·high · Fable 5·medium · Fable 5·low · Opus 4.8·max · Opus 4.7·max · Haiku 5.5·max · Sonnet 5·max · Sonnet 5·xhigh · Haiku 5.5·xhigh · Sonnet 5·high · Sonnet 4.6·high · Haiku 5.5·high · Haiku 5.5·medium · Sonnet 5·medium · Haiku 5.5·low · Sonnet 5·low
Conceptual Reasoning Index · LMCA
conceptualreasoning.ai
Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 5.5·xhigh · Opus 5.5·medium · Fable 5.1·medium · Fable 5.1·max · Opus 5·xhigh · Opus 5.5·low · Fable 5.1·xhigh · Opus 5·high · Sonnet 5.5·medium · Fable 5.1·high · Sonnet 5.5·high · Opus 5·max · Sonnet 5.5·low · Fable 5·max · Opus 5·medium · Fable 5.1·low · Fable 5·xhigh · Opus 5·low · Fable 5·high · Fable 5·medium · Fable 5·low · Opus 4.8·max · Opus 4.7·max · Haiku 5.5·max · Sonnet 5·max · Sonnet 5·xhigh · Haiku 5.5·xhigh · Sonnet 5·high · Sonnet 4.6·high · Haiku 5.5·high · Haiku 5.5·medium · Sonnet 5·medium · Haiku 5.5·low · Sonnet 5·low
Convex evals (no guidelines, web research)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines, web research' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelOpus 5.5·medium · Fable 5.1·medium · Opus 5·medium · Sonnet 5.5·medium · Opus 4.8·medium · Sonnet 5·medium
Convex evals (no guidelines)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelOpus 5.5·medium · Fable 5·medium · Sonnet 5.5·medium · Opus 4.8·medium · Opus 5·medium · Fable 5.1·medium · Sonnet 5·medium
Convex evals (with guidelines)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'with guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelSonnet 5.5·medium · Fable 5·medium · Fable 5.1·medium · Opus 5.5·medium · Opus 5·medium · Opus 4.8·medium · Sonnet 5·medium
Convex LLM leaderboard v92a60e0b · no guidelines
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines; read 2026-10-15 ✓xmodelOpus 5.5·medium · Fable 5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Opus 4.8·medium · Sonnet 5·medium · Haiku 5.5·medium
Convex LLM leaderboard v92a60e0b · no guidelines + web
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines + web; read 2026-10-15 ✓xmodelOpus 5.5·medium · Fable 5·medium · Fable 5.1·medium · Sonnet 5.5·medium · Opus 4.8·medium · Opus 5·medium · Sonnet 5·medium · Haiku 5.5·medium
Convex LLM leaderboard v92a60e0b · with guidelines
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, with guidelines; read 2026-10-15 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Fable 5·medium · Fable 5.1·medium · Opus 5·medium · Opus 4.8·medium · Sonnet 5·medium · Haiku 5.5·medium
costrouter
github-costrouter
—xmodelOpus 5·low
CRBugBench post-merge bugs (Fix track), recall@5
arxiv-2610.13668
500 review tasks per track from 98 OSS repos, copilot-cli 1.0.74 official /review prompt, default prompt; Table 1 of arXiv 2610.13668; read 2026-10-15 ✓xmodelOpus 5·xhigh · Opus 4.8·xhigh · Sonnet 5·xhigh · Haiku 4.5·nothink
CRBugBench pre-merge bugs (Cmt track), recall@5
arxiv-2610.13668
500 review tasks per track from 98 OSS repos, copilot-cli 1.0.74 official /review prompt, default prompt; Table 1 of arXiv 2610.13668; read 2026-10-15 ✓xmodelOpus 5·xhigh · Opus 4.8·xhigh · Sonnet 5·xhigh · Haiku 4.5·nothink
ctala
github-ctala
Claude Code CLI · reasoning not configured, temp 0.7 ✓xmodelHaiku 5.5·nothink · Sonnet 5.5·high · Opus 5.5·high · Fable 5.1·high · Opus 5·high · Haiku 4.5·nothink
CTI-REALM unconstrained (300-message budget): overall reward
ctirealm-2610.14180
Unconstrained CTI-REALM (Averlon), Table 2: 50 tasks, one sample each, common harness, judges Gemma-4-31B thinking / DeepSeek-v4-flash / GLM-5.3-flash x3 trials, message budget 300 (14 models have samples run at 300 incl. 10 spliced, 12 carried from lower budgets with zero capped samples); overall weighted reward; checkpoints C0-C4 same runs (rule 8); cost = cost of the 50-task run at list prices; arXiv 2610.14180 (Oct 2026); read 2026-10-20 ✓xmodelOpus 5·high · Haiku 4.5·nothink
CUA-SWE mobile hybrid: task success %
cua-swe-2609.32600
CUA-SWE Table 8: single attempt per task, mobile domain (20 tasks), deterministic executable verifiers, access = hybrid CUA (screenshots + graphical actions); frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓xmodelOpus 4.8·high · Sonnet 5·high · Fable 5·high
CWE-bench (120 vulnerability-repair tasks, 4 attempts)
arxiv-2610.17020
Claude Code, high reasoning effort, 4 attempts per task, 1 h limit; programmatic pass@1; cost = mean API cost per rollout; read 2026-10-15 ✓xmodelOpus 5.5·high · Fable 5.1·high
cwe-bench-v2-class-k01
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k02
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k03
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k04
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k05
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k06
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k07
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k08
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k09
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-class-k10
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-combined-pass1
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-combined-pass4
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-language-c-cpp
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-language-go
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-language-java
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-language-js-ts
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-language-python
cwe-bench
—xmodelOpus 5.5·high
cwe-bench-v2-language-rust
cwe-bench
—xmodelOpus 5.5·high
Data Science Tasks (Internal) (OpenAI GPT-6 Astra post chart)
openai
Data Science Tasks (Internal); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5.1·max · Opus 5·max · Fable 5·max
DataLLM Lab: 9 executed Python tasks, Qwen3.8 Max family (Prime, Max, Max-0902), pass and cost per 1,000 tasks
datallmlab
9 executed Python tasks (python3 -I subprocess, 12 s timeout), temperature 0, max_tokens 4000, one scored attempt, via OpenRouter; three runs on different days (Prime 2026-10-02, Max 2026-08-20, Max-0902 2026-09-16); cost = measured tokens x list price on the priced-on date (source-computed, 'not a billing statement'); read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 5·high · Opus 4.7·nothink · Fable 5.1·high · Opus 5.5·high
Dealwatch defect review
github-dealwatch
Claude Code orchestrator · 5 re-seeded bugs, Fable 5.1 vs Opus 5.5 @high, 1 run, list-price cost ✓xmodelFable 5.1·high · Opus 5.5·high
Dealwatch small feature
github-dealwatch
same repo · 6-req feature, hidden tests saturated → cost only ✓xmodelFable 5.1·high · Opus 5.5·high
deepseek-v41-techreport: ale
deepseek
Agents' Last Exam (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold(ALE-CLI); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: autobench
deepseek
AutomationBench v1.0.6 public set (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: babyvision
deepseek
BabyVision w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: chartography
deepseek
Chartography w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: exploitgym
deepseek
ExploitGym (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: hle
deepseek
HLE (Pass@1), full set; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: hle-tools
deepseek
HLE with tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: nl2repo
deepseek
NL2Repo-Bench (Score); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: programbench
deepseek
ProgramBench (Almost@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: tb30
deepseek
Terminal-Bench 3.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: tb40
deepseek
Terminal-Bench 4.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
deepseek-v41-techreport: zerobench
deepseek
ZeroBench-main w/ tools (Pass@5); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelOpus 5·max
DeepSWE (OpenAI GPT-6 Sol and Luna post)
openai
DeepSWE; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
DeepSWE 1.0 (xAI Grok 4.5 post)
xai
DeepSWE 1.0 pass@1 %; footnote Eval created by Datacurve, run with each model provider's harnesses by AA; xAI post 16 Jul 2026; read 2026-10-19 ✓xmodelOpus 4.8·max · Opus 4.7·max
DeepSWE v1.1 (OpenAI GPT-5.6 post chart)
openai
DeepSWE v1.1 pass@1, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max
DeepSWE v1.1 (xAI Grok 4.7 post)
xai
DeepSWE v1.1, resolve rate %, harness/n/attempts not stated; xAI Grok 4.7 launch post 21 Sep 2026; read 2026-10-19 ✓xmodelFable 5.1·max
deepu-cc-set1-of8
deepu
—xmodelOpus 5.5·xhigh · Opus 5.5·medium
deepu-cc-set2-of8
deepu
—xmodelOpus 5.5·xhigh · Opus 5.5·medium
Deloitte code review
anthropic-partner-deloitte
Opus 5.5 launch page partner quote · bugs caught, score only, n unstated ✓xmodelOpus 5.5·low · Opus 5·high
dentsu-b
dentsusoken
—xmodelSonnet 5·high · Opus 5·high · Fable 5·high
dentsu-d
dentsusoken
—xmodelSonnet 5·high · Opus 5·high · Fable 5·high
Design Creativity Bench (UI generation; overall T-score, median cost per design)
arxiv-2610.11539
80 UI design tasks / 160 prompts, one generation per design, highest effort (Opus 5.5 at xhigh); Tables 2 and 9 of arXiv 2610.11539; read 2026-10-15 ✓xmodelOpus 5.5·xhigh · Fable 5.1·max
Design Tasks (Internal) (OpenAI GPT-6 Astra post chart)
openai
Design Tasks (Internal); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5.1·max
DevelopersIO Project Euler 1012 (hard), Claude 5 family + Haiku 5.5 effort ladder
classmethod
claude -p --effort medium (trio) / Haiku 5.5 five efforts x 2 runs; Bash/Read/Write/Edit/WebFetch; cost = Claude Code estimate; read 2026-10-15 ✓xmodelOpus 5.5·medium · Fable 5.1·medium
DexHoldem (2605.18727v2) — 36 agentic perception problems, Claude Code, avg of 3 runs
arxiv-2605.18727
Table 2, overall strict % and per-category average column; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelOpus 5.5·medium · Sonnet 5·medium
dexholdem-overall
arxiv-2605.18727
—xmodelOpus 5.5·medium · Sonnet 5·medium
digirise/チャエン: 3 prompts (game/summary/earnings), total time/cost, n=1
digirise-chaen
digirise.ai/chaen-ai-lab/claude-opus-5-5/ (cites チャエン's X post 2026-09-23); Claude Code, 1 run per model per prompt, 3 prompts summed; effort-from-config: "Claudeの2モデルはClaude Code経由でeffortはxhigh"; read 2026-10-01 ✓xmodelOpus 5.5·xhigh · Fable 5.1·xhigh
Double-pendulum job, Haiku 5.5 effort ladder + Sonnet 5.5 medium (AI Receipts video)
youtube-aireceipts
3 runs each; pass = energy drift < 0.1% at 2 simulated min; cost = exact token counts x list price; 2026-10-08; read 2026-10-18 ✓xmodelHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max
DrivingBench: progress along a 127 m real-car cone course (best attempt, %)
drivingbench-2609.38948
DrivingBench Table 2: VLM drives a real Toyota Corolla through 3 tools, one chat session per model = one trial (n=1), up to 3 attempts with retained context and a fixed reflection prompt; progress % = furthest point on the centreline (best attempt); each model in its native harness (Codex CLI 0.154.0-alpha.6.2 for GPT-6 Astra and GPT-5.6 Sol; Claude Code 2.1.274; Cursor); cost = tokens x public list prices as of Sep 2026 summed over all attempts; values typed by hand from the page-7 image (the text layer runs digits together); arXiv 2609.38948 (Sep 2026); read 2026-10-20 ✓xmodelFable 5.1·medium
Dyad app-builder · Deskhero
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Deskhero (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Fable 5·medium · Sonnet 5·medium
Dyad app-builder · Portalis
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Portalis (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Fable 5·medium · Sonnet 5·medium
Dyad app-builder · Relay CRM
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Relay CRM (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Fable 5·medium · Sonnet 5·medium
E-Bench (MCP tools): Avg@3 task success
ebench-2607.23722
E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench (MCP tools): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓xmodelOpus 4.8·max
E-Bench-Code (code execution): Avg@3 task success
ebench-2607.23722
E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench-Code (code execution): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓xmodelOpus 4.8·max
E-Commerce Bench (arXiv 2608.30730)
arxiv-2608.30730
E-Commerce Bench, simulated e-commerce store, 365-day task, 18 tools, 4000-turn cap, 5 independent episodes per model in one fixed world, harness = the paper's agent loop; score = mean end-of-year total assets (kilo-yuan) from a 100k stake, bankrupt episodes kept in the mean; no cost reported; arXiv v1 2026-08-31; read 2026-10-20 ✓xmodelFable 5·max · Opus 4.8·max · Opus 4.7·max
econ-eval leaderboard A (20 tasks, Opus-5 judge)
github-deluair-econ-eval
deluair/econ-eval README at commit 77f9f3e (16 Oct 2026), leaderboard of 2026-09-10: 20 economics tasks (quantitative, reasoning, coding, writing) x 5 samples = 100 completions/model, numeric/exact/code-exec graders + rubric tasks judged by Claude Opus 5; score = mean over tasks of task means; cost = total for the 100 completions from token counts x list price (prices read from the OpenRouter catalog 2026-07-31 / vendor pages); per-track tables are the same completions (not extra groups); ten models ran 2026-08-01, deepseek-flash added 2026-09-10 same tasks/samples/judge; read 2026-10-20 ✓xmodelOpus 4.8·high
EEBench (xAI Grok 4.7 post)
xai
EEBench electrical engineering accuracy %, harness/n not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓xmodelFable 5.1·max
EMB
vals.ai
Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Sonnet 5.5·max · Haiku 5.5·max
energy-before-only
arxiv-2610.26538
—xmodelHaiku 5.5·low · Sonnet 5.5·low · Opus 5.5·low
energy-both-photos
arxiv-2610.26538
—xmodelHaiku 5.5·low · Sonnet 5.5·low · Opus 5.5·low
ENSO stochastic-model building (SciExam), final composite score
arxiv-2610.10513
agent builds an ENSO model within 6 h eligible time, one run per system, Claude Code on subscription; Table 4/6 of arXiv 2610.10513; read 2026-10-15 ✓xmodelFable 5.1·medium · Opus 5·medium
Epoch · chess puzzles
epoch.ai
Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelSonnet 4.6·high · Opus 4.7·max · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.8·max · Opus 4.8·low · Fable 5·max · Fable 5·high · Fable 5·low · Sonnet 5·max · Sonnet 5·xhigh · Opus 5·max · Opus 5·low · Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Haiku 5.5·low · Fable 5.2·max
Epoch · ebr bench
epoch.ai
Epoch AI's own evaluation (ebr_bench), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Opus 4.8·max · Fable 5·max · Opus 4.7·max · Fable 5.2·max
Epoch · frontiermath
epoch.ai
Epoch AI's own evaluation (frontiermath), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
Epoch · frontiermath erdos
epoch.ai
Epoch AI's own evaluation (frontiermath_erdos), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelSonnet 5.5·max · Opus 5.5·max
Epoch · frontiermath tier 4
epoch.ai
Epoch AI's own evaluation (frontiermath_tier_4), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
Epoch · frontiermath tier 4 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max · Fable 5·max
Epoch · frontiermath tiers 1 3 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max · Fable 5·max · Fable 5.2·max
Epoch · furniture assembly
epoch.ai
Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Opus 4.7·max · Fable 5.2·max
Epoch · gpqa diamond
epoch.ai
Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Sonnet 5·max · Opus 5·low · Opus 4.8·low · Opus 4.7·max · Fable 5·max · Fable 5·low · Fable 5·high · Opus 5·max · Sonnet 4.6·high · Sonnet 5·xhigh · Opus 4.8·max · Opus 4.7·xhigh · Fable 5.2·max
Epoch · mirrorcode
epoch.ai
Epoch AI's own evaluation (mirrorcode), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelOpus 5.5·max · Opus 5.5·high · Fable 5.1·high · Opus 4.7·high · Fable 5·high
Epoch · mystery game puzzles
epoch.ai
Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Sonnet 4.6·low · Fable 5·max · Sonnet 5·max · Opus 4.8·xhigh · Opus 4.8·max · Opus 5·max · Opus 4.7·max · Fable 5.2·max
Epoch · otis mock aime 2024 2025
epoch.ai
Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·xhigh · Haiku 5.5·high · Sonnet 5·max · Opus 5·low · Opus 4.8·low · Opus 4.7·max · Fable 5·low · Opus 5·max · Sonnet 4.6·high · Sonnet 5·xhigh · Fable 5·max · Opus 4.8·max · Opus 4.7·xhigh · Fable 5.2·max
Epoch · simpleqa verified
epoch.ai
Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.7·xhigh · Opus 4.8·max · Opus 5·max · Fable 5·xhigh · Sonnet 4.6·high · Fable 5.2·max
Epoch · swe bench verified
epoch.ai
Epoch AI's own evaluation (swe_bench_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelOpus 4.7·max
Epoch AI HLE external table (accuracy %, 2026-10-15 zip)
epoch.ai
epoch.ai/data/benchmark_data.zip (hle_external.csv; zip read 2026-10-15; copy gc/c4/src/epoch/2026-10-15); Humanity's Last Exam, accuracy with standard error; Notes column empty (provenance unstated); values differ from every Scale HLE capture (not a restatement); read 2026-10-20 ✓xmodelFable 5.1·xhigh · Sonnet 5.5·xhigh
epoch-ext-btf3
epoch.ai
—xmodelOpus 4.8·xhigh · Fable 5·high · Opus 4.8·high · Sonnet 5·xhigh · Opus 5·xhigh
epoch-ext-deepresearchbench
epoch.ai
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium
epoch-ext-enigma-eval
epoch.ai
—xmodelFable 5·high · Opus 4.8·xhigh
epoch-ext-surface-evolver-bench
epoch.ai
—xmodelFable 5·high · Opus 4.8·high · Sonnet 5·medium
epoch-spireclimb-sts1-a20-heart
epoch.ai
—xmodelOpus 5.5·max · Fable 5.2·max
epoch-spireclimb-sts2-a10
epoch.ai
—xmodelSonnet 5.5·max · Fable 5.1·max · Opus 5.5·max · Fable 5.2·max
eval-forge-lost-temple-P
github-eval-forge
—xmodelOpus 5.5·max · Fable 5.2·xhigh · Opus 5.5·xhigh · Fable 5.2·high · Opus 5.5·high · Opus 5.5·medium
eval-forge-lost-temple-S
github-eval-forge
—xmodelOpus 5.5·max · Fable 5.2·xhigh · Opus 5.5·xhigh · Fable 5.2·high · Opus 5.5·high · Opus 5.5·medium
eval-forge-lost-temple-sxp
github-eval-forge
—xmodelOpus 5.5·max · Fable 5.2·xhigh · Opus 5.5·xhigh · Fable 5.2·high · Opus 5.5·high · Opus 5.5·medium
EvoClaw
evo-claw.com
EvoClaw, long-horizon repository evolution in Claude Code, current board v1.0.2 · overall score in %, total cost of the run (USD) and output tokens; effort from the board's label (Opus 4.8 max 1M, Opus 4.7 xhigh; none stated otherwise); the Opus 4.7 1M-context and OpenHands rows left out (another configuration); read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
fcode20pass
cognition
—xmodelFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
fcode20score
cognition
—xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
fcodev1ext
cognition
—xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·xhigh
FinancialAuditBench (2609.32835) — 45 audit workpaper tasks, bash-only agent, 8 runs/task
arxiv-2609.32835
Table 8/Table 9 main row, strict pass % and mean cost $/trajectory, 360 trajectories/model; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelOpus 5.5·medium · Fable 5.1·medium
Firecrawl 57-run
firecrawl
claude -p in sandbox-exec · 3 models at matched high, measured usage; 7/7 all → cost only ✓xmodelFable 5·high · Fable 5.1·high · Opus 5·high
fireworks-dsflash-deepswe
fireworks-blog
—xmodelOpus 5·max
FlavourBench Score (mean of 3 families x 178 tasks)
flavourbench-2608.20574
FlavourBench Table 3 leaderboard, 534 tasks, FB Score = equal-family mean, 2 collection panels pooled, OpenRouter fixed provider (GLM 5.3 direct); arXiv 2608.20574 (20 Aug 2026); family columns Table 10 are same runs (rule 8); read 2026-10-20 ✓xmodelFable 5·low
ForecastBench preliminary leaderboard: dataset-question Brier Index, web search + code execution, 128k, 984 questions
forecastbench
ForecastBench preliminary leaderboard; dataset score = difficulty-adjusted Brier Index; N=984 resolved dataset questions; cost absent; read 2026-10-18 ✓xmodelOpus 5·max · Fable 5·max · Fable 5·high
forecastbench-tourn-ws128k-overall
forecastbench
—xmodelOpus 5·max · Fable 5·max
FrontierCode (OpenAI GPT-6 Sol and Luna post)
openai
FrontierCode; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓xmodelFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
FrontierCode 1.1 Extended (OpenAI GPT-6 Astra post chart, code mode)
openai
FrontierCode 1.1 Extended; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = code; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max
FrontierCode 1.1 Extended (OpenAI GPT-6 Astra post chart, non-code mode)
openai
FrontierCode 1.1 Extended; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
FrontierCode v1.1 Extended (xAI Grok 4.6 post)
xai
FrontierCode v1.1 (Extended) %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓xmodelFable 5·max
FrontierFinance T4
arxiv-2608.11683
arXiv 2608.11683 Table 4 · effort omitted, Fable 5 vs Opus 4.8 ✓xmodelFable 5·high
FrontierMath Tier 4 (v2) (OpenAI GPT-6 Astra post chart, non-code mode)
openai
FrontierMath Tier 4 (v2); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
FrontierSWE dominance score (GLM-5.2 card, as of 2026/06/16)
zhipu
FrontierSWE, evaluated by Proximal, 1M context 128K output, dominance score as of 2026/06/16; held group frontierswe is mean-reward@5 (other metric and date); read 2026-10-19 ✓xmodelOpus 4.8·max
FrontierSWE dominance score (GLM-5.3 card, as of 2026/08/14)
zhipu
FrontierSWE, evaluated by Proximal, 1M context, 128K output, dominance score as of 2026/08/14; held group frontierswe is mean-reward@5 (glm-5.3 30.18 there vs 78.1 here: different metric); read 2026-10-19 ✓xmodelOpus 4.8·max · Fable 5·max
FrontierSWE v2
frontierswe
Proximal frontierswe.com + Sonnet 5.5 card · 34 ultra-long tasks, max effort, proximus harness, mean reward@5; cost basis undocumented ✓xmodelFable 5.1·max
GameLogicBench
arxiv-2609.21562
arXiv 2609.21562 · 72 Godot game-logic tasks in Claude Code, effort=high, Opus 5 vs Sonnet 5, tokens × vendor list ✓xmodelOpus 5·high · Sonnet 5·high
garrytan/gbrain-evals N2 judge replay (6 Oct 2026): 741 judged pairs, contradiction detection prompt v4; recall saturated 150/150 for all three, cost-only
github-gbrain-evals
docs/benchmarks/2026-10-06-n2-judges; cost from the repo budget ledger (741 paid requests); no effort control; read 2026-10-18 ✓xmodelFable 5.1·high
GDPval (Elo) (xAI Grok 4.7 post)
xai
GDPval Elo (Artificial Analysis GDPval-AA family by Elo scale); chart only; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓xmodelFable 5.1·max
GDPval-AA v2 (OpenAI GPT-5.6 post chart)
openai
GDPval-AA v2; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
GDPVal-AA v2 (xAI Grok 4.6 post)
xai
GDPVal-AA v2 Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓xmodelFable 5·max
gdpx-overall
semianalysis
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max · Haiku 5.5·max
gdpx-p-arcverex
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-aria
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-arithmetic
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-beautiful-machines
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-chakra
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-datacurve
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-emulated
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-ergodex
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-extrapolate
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-fleet
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-latchbio
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-living-context
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-third-moment
semianalysis
—xmodelHaiku 5.5·max
gdpx-p-valenceforge
semianalysis
—xmodelHaiku 5.5·max
GM-Bench 2.0 agentic lane (hockey-GM 5-season episode, 32 private seeds, Claude Code)
github-gmbench
nedcut/gm-bench README (rows published 12-15 Oct 2026); Claude Code harness default effort; mean score over 32-seed private panel, reference pick-trader heuristic = 249.2; rows flagged unpinned by the source; read 2026-10-15 ✓xmodelSonnet 5.5·medium · Opus 5.5·medium
GM-Bench 2.1: 5-season hockey GM inside Claude Code, 32-seed private panel, score-v2 (released 2026-10-16)
github-gmbench
nedcut/gm-bench results/agentic_v21 claude-2.1.281 panel-32x5; reference pick-trader 249.618 (2.1 sim-v4-league21); NEW GROUP: 2.1 is a new simulator/panel, held gm-bench2-agentic is 2.0 (pick-trader 250.068); read 2026-10-18 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
GRIP closed-loop accuracy (E1, mean over Q1-Q5), spatial and physical reasoning for VLMs
grip-2610.16170
GRIP Table 3 Mean (E1): 34 domains, procedurally generated images with closed-form answers, 5 questions per image in fixed order, fresh single-turn per item, 8,500 queries per hosted model, routed API; hosted max output 256-2048 tokens; Q1-Q5 columns are the same runs (rule 8); arXiv 2610.16170 (Oct 2026); read 2026-10-20 ✓xmodelOpus 5·low · Sonnet 5·low
habr-crashtest27
habr-author
—xmodelOpus 5.5·high
Haiku 5.5 launch page: Terminal-Bench 4.0, cache-read price-cut illustration (7 Oct 2026)
anthropic
anthropic.com/claude-haiku-5-5 launch page, 'Further updates' section, embedded chart CSV (exact); read 2026-10-08 ✓xmodelHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
Haiku 5.5 system card: biomedical image analysis, max effort
anthropic
CT/MRI/histopathology/dermoscopy/retinal/ultrasound/microscopy imaging, scaled to best public-competition result; 1 attempt/problem (exception to the chapter default); read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5·max
Haiku 5.5 system card: FrontierSWE v2, Proximal's own harness, max effort
anthropic
34 ultra-long-horizon tasks, 20h trial budget, mean of 5 trials/task, Proximal's own harness; read 2026-10-08 ✓xmodelHaiku 5.5·max · Fable 5·max
Haiku 5.5 system card: Humanity's Last Exam (with tools) score+cost by effort, 5 efforts
anthropic
chart image, y-axis printed score labels per point; x-axis (cost, log scale) has printed tick labels ($0.01,$0.02,$0.03,$0.05,$0.07,$0.10,$0.20,$0.30,$0.50,$0.70,$1,$2) but no per-point numeric cost label, so cost is digitised from the marker pixel position (calibration in notes.md: x=1685.88+709.69*log10(cost), fit from the $0.01 and $1 ticks, cross-checked exact at $0.1); Haiku 4.5's single point's x could not be confidently isolated from the crowded pink-diamond legend swatch vs. data point in the time available, left without cost; read 2026-10-08 ✓xmodelHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
Haiku 5.5 system card: LatchBio Bioinformatics, SingleCellBench, max effort
anthropic
single-cell RNA sequencing analysis; read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5·max
Haiku 5.5 system card: LatchBio Bioinformatics, SpatialBench Verified, max effort
anthropic
spatial transcriptomics / chromatin-accessibility analysis; bash tool, file editor, prespecified packages; read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5·max
Haiku 5.5 system card: medicinal chemistry (ADME), max effort, xmodel
anthropic
504 questions, drug-like molecule structures, ADME property comparison; read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5·max · Opus 5.5·max · Sonnet 5.5·max
Haiku 5.5 system card: morphology-to-molecule matching (Axiom Bio), max effort
anthropic
matches 4 multi-channel cell-imaging sets to 4 shuffled molecule-dose perturbations, human liver in-vitro assay; read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5.5·max
Haiku 5.5 system card: protein design, Library Ranking, max effort
anthropic
prioritize protein designs for experimental testing; 3 attempts/problem (exception to the chapter default); read 2026-10-08 ✓xmodelSonnet 5·max · Sonnet 5.5·max
Haiku 5.5 system card: protein design, Sequence Generation, max effort
anthropic
novel protein sequence generation under design constraints; 1 attempt/problem (exception to the chapter's 5-attempt default, stated explicitly); read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5·max · Sonnet 5.5·max
Haiku 5.5 system card: Protocols, Troubleshooting, max effort
anthropic
detect and fix issues in molecular biology lab protocols; read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5·max · Sonnet 5.5·max
Haiku 5.5 system card: Protocols, Understanding (V2, Benchling), max effort
anthropic
real lab protocol document + practical application question; read 2026-10-08 ✓xmodelSonnet 5·max · Sonnet 5.5·max
Haiku 5.5 system card: Terminal-Bench 4.0, safeguards-on, each model's own best-effort config
anthropic
Claude Code --bare mode; Haiku 5.5/Haiku 4.5 10 trials/task (660), Sonnet/Opus 5.5 5 trials/task (330); safeguard-flagged trials counted as failures; read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·xhigh
Haiku 5.5 system card: Terminal-Bench-Science 0.1, safeguards-on, max effort
anthropic
Claude Code --bare mode, max thinking effort for all except Haiku 4.5 (fixed budget); 700 trials for Haiku 5.5/4.5; read 2026-10-08 ✓xmodelHaiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max
Haiku 5.5 system card: WANDR soft F1 score+cost by effort, 5 efforts
anthropic
chart image, y-axis printed score labels per point; x-axis (cost, log scale) ticks $0.005,$0.01,$0.02,$0.05,$0.10,$0.20,$0.50,$1,$2,$5,$10,$20 give calibration x=1230.9+457.94*log10(cost) (fit from $0.005 and $20 ticks, cross-checked exact at $1 and $0.10); several Opus/Sonnet 'Med'/'High' markers were partly occluded by a crossing line of another series and their centroid is a merge of fragment pixels (lower confidence, noted in notes.md); Haiku 4.5's point not confidently isolated, left without cost; read 2026-10-08 ✓xmodelHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
hama-jp: coastal Viking village (voxel)
youtube-hama-jp
Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓xmodelHaiku 5.5·high · Sonnet 5.5·high
hama-jp: five-story pagoda garden (three.js)
youtube-hama-jp
Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓xmodelHaiku 5.5·high · Sonnet 5.5·high
hama-jp: model railway with interlocking (3h cap)
youtube-hama-jp
Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓xmodelHaiku 5.5·high · Sonnet 5.5·high
Harbor-Index 1.0 (82 tasks), Terminus-2 harness for all models
arxiv-2609.04298
Harbor-Index 1.0: 82 tasks selected from 54 Harbor benchmarks, 3 trials, harness Terminus-2 for every model; pass rate % and approximate per-run cost (USD) of one full Index run; provider default settings (paper: reasoning effort at provider default); models other than GPT-5.5/Opus 4.8/Gemini 3.1 Pro served via OpenRouter; read 2026-10-20 ✓xmodelOpus 4.8·nothink
Harbor-Index 1.0 (82 tasks), vendor-native harness (Codex / Claude Code / Gemini CLI; Claude Code for open models)
arxiv-2609.04298
Harbor-Index 1.0: 82 tasks, 3 trials; harness = vendor-native (Codex for GPT, Claude Code for Claude, Gemini CLI for Gemini, Claude Code for six OpenRouter-served models); pass rate % and approximate per-run cost (USD); read 2026-10-20 ✓xmodelOpus 4.8·high
Harbor-Index adapter AA-LCR, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter AA-LCR evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter Aider Polyglot, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Aider Polyglot evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter AIME 24 & 25, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter AIME 24 & 25 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter AlgoTune, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter AlgoTune evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter ARC-AGI-2, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter ARC-AGI-2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter BFCL, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter BFCL evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter BigCodeBench-Hard, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter BigCodeBench-Hard evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter BIX-Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter BIX-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter CodePDE, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CodePDE evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter CompileBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CompileBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter CRUST-Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CRUST-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter CyberGym, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CyberGym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter DA-Code, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter DA-Code evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter DeepSynth, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter DeepSynth evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter FeatureBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter FeatureBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter FinanceAgent, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter FinanceAgent evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter GAIA, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GAIA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter GAIA2, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GAIA2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter GPQA Diamond, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GPQA Diamond evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter GSO, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GSO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter HumanEvalFix, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter HumanEvalFix evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink
Harbor-Index adapter Humanity’s Last Exam, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Humanity’s Last Exam evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter IneqMath, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter IneqMath evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter KUMO, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter KUMO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter LAB-Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter LAB-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter LawBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter LawBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter LiveCodeBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter LiveCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter MedAgentBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter MedAgentBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter MMAU, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter MMAU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter MMMLU, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter MMMLU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter Omni-Math, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Omni-Math evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter PIXIU, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter PIXIU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter QCircuitBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter QCircuitBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter QuixBugs, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter QuixBugs evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter Reasoning Gym, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Reasoning Gym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter ReplicationBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter ReplicationBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter ResearchCodeBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter ResearchCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SciCode, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SciCode evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter Seal-0, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Seal-0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SimpleQA, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SimpleQA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SkillsBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SkillsBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SLDBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SLDBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter Spider 2, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Spider 2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SpreadsheetBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SpreadsheetBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter StrongReject, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter StrongReject evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SWE-Bench Pro, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-Bench Pro evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SWE-bench Verified, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-bench Verified evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SWE-Bench-Multilingual, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-Bench-Multilingual evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SWE-Lancer, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-Lancer evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SWE-smith, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-smith evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter SWT Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWT Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter Terminal-Bench 2.0, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Terminal-Bench 2.0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter USACO, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter USACO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harbor-Index adapter WideSearch, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter WideSearch evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink
Harvey Legal Agent Benchmark (xAI Grok 4.7 post)
xai
Harvey Legal Agent Benchmark %, harness/judge not stated; xAI Grok 4.7 post 21 Sep 2026 (4.6 post calls it 'Harvey LAB (Vals)'); read 2026-10-19 ✓xmodelFable 5.1·max
HealthBench Professional (OpenAI GPT-6 Astra post chart, non-code mode)
openai
HealthBench Professional; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·xhigh · Opus 5·max
HealthBench Professional (xAI Grok 4.7 post)
xai
HealthBench Professional % (clinical reasoning), grader/variant not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓xmodelFable 5.1·max
hehee9 Maze Bench Tier 1 (30 mazes)
hehee9maze-bcd609c
Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 30 mazes (6 sizes x 5, 4x4..18x18), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status completed); read 2026-10-20 ✓xmodelFable 5·medium · Fable 5.1·medium · Haiku 5.5·medium · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium
hehee9 Maze Bench Tier 2 (20 mazes)
hehee9maze-bcd609c
Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 20 mazes (15x15..24x24, 5 each), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status running); read 2026-10-20 ✓xmodelFable 5.1·medium · Opus 5.5·medium · Sonnet 5.5·medium
Hex DataBench
hex.tech
Hex DataBench, 100 analytics tasks in the Hex workspace, rubric-judged · score in %, average cost per task (USD), tokens per task (one total); effort from the row label; Opus 5, Fable 5.1, Fable 5, Opus 4.8, Sonnet 5 at every listed effort (Opus 5.5 in its own group); read 2026-10-01 ✓xmodelOpus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Fable 5.1·max · Haiku 5.5·low · Fable 5.1·xhigh · Fable 5.1·medium · Fable 5.1·high · Opus 5·max · Opus 4.8·xhigh · Opus 5·high · Fable 5·max · Opus 5·xhigh · Opus 4.8·medium · Opus 5·medium · Fable 5.1·low · Fable 5·high · Opus 5·low · Opus 4.8·max · Fable 5·low · Fable 5·medium · Sonnet 5·xhigh · Opus 4.8·high · Sonnet 5·high · Sonnet 5·medium · Sonnet 5·max · Opus 4.8·low · Sonnet 5·low
heydonna-rt-v1.1-text-legal-structure
heydonna
—xmodelSonnet 5.5·high
hf-kimi-k3.1-pro-aa-briefcase
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-ale-all
moonshot
—xmodelOpus 5.5·max
hf-kimi-k3.1-pro-ale-cli
moonshot
—xmodelOpus 5.5·max
hf-kimi-k3.1-pro-apex-agents
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-automationbench-aa
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-babyvision-py
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-benchcad-notools
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-benchcad-tools
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-biomystery
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-browsecomp
moonshot
—xmodelOpus 5.5·max
hf-kimi-k3.1-pro-business-operation
moonshot
—xmodelFable 5.1·max
hf-kimi-k3.1-pro-chartography-py
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-charxiv
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-charxiv-py
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-deck-arena
moonshot
—xmodelOpus 5.5·max
hf-kimi-k3.1-pro-deepswe
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-design-bench
moonshot
—xmodelOpus 5.5·max
hf-kimi-k3.1-pro-exploitbench
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-exploitgym
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-frontier-finance
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-gdppdf
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-gdppdf-py
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-gdpval-aa
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-gpqa-diamond
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-hallu-hard-tools
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-hallu-notools
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-handbook
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-harvey-legal
moonshot
—xmodelFable 5.1·max
hf-kimi-k3.1-pro-hle-diamond
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-hle-full
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-hle-tools
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-kcb3
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-mmmupro
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-mmmupro-py
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-officeqa-pro
moonshot
—xmodelFable 5.1·max
hf-kimi-k3.1-pro-officeqa-pro-v2
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-osworld-v21
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-perceptionbench
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-posttrainbench
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-programbench
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-swemarathon
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-tb4
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-tbs
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-zerobench-avg5
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-zerobench-pass5
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hf-kimi-k3.1-pro-zerobench-py-pass5
moonshot
—xmodelFable 5.1·max · Opus 5.5·max
hotkv-banking77-fresh
hotkv
—xmodelFable 5.1·low · Opus 5.5·low
HouseholdBench: household-economics prediction, macro-F1 on categorical targets
arxiv-2610.07563
32 household tasks, pre-cutoff test split, default reasoning effort; Table 3 of arXiv 2610.07563; read 2026-10-15 ✓xmodelFable 5.1·high · Opus 5·high · Opus 4.8·nothink
Humanity's Sixth Sense (HSS), single-pass API, pass@1
arxiv-2610.08966
522 image/video tasks, pass@1 = every rubric criterion met, effort = highest honoured per Table 5; tokens_out = mean thinking + answer tokens per task; read 2026-10-15 ✓xmodelOpus 5.5·xhigh · Fable 5.1·xhigh · Fable 5·max · Opus 5·max · Sonnet 5·max · Opus 4.8·max
Hy-MultiTurn (Chinese deep multi-turn dialogue understanding): Importance score
hymultiturn-2607.29196
Hy-MultiTurn leaderboard (Tencent Hunyuan authors; Hy3/Doubao rows are not in the eight providers): six dimensions, final-turn response scored against weighted rubric items; Importance (weighted correct share) = primary metric, mean of three independent rolls; arXiv 2607.29196 (31 Jul 2026); read 2026-10-20 ✓xmodelOpus 4.7·high
hypfrontier-challenge64-hf-valid
hypfrontier-2608.10843
—xmodelFable 5·medium
iam_lima: weekly-report-from-notes task, mean cost/time of 3 runs
note-iam_lima
note.com/iam_lima/n/ncfed4a80e005; Claude Code 2.1.284, work-notes with a planted weekday error and a planted arithmetic error, 3 runs per setting, means reported directly by the source; effort-from-config: "effortはAPIの既定にあたるhighにそろえ、Sonnet 5.5だけはClaude Codeの既定のmediumでも回しました"; read 2026-10-01 ✓xmodelOpus 5.5·high · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5·high
ibmartian-frontier10-p-claude
ironbee
—xmodelFable 5.1·low · Fable 5.1·high · Fable 5.1·xhigh
ibmartian-full-p-claude
ironbee
—xmodelSonnet 5.5·low · Opus 5.5·low · Opus 5.5·medium · Opus 5·low · Sonnet 5.5·high · Sonnet 5.5·xhigh · Opus 5.5·xhigh · Opus 5·high · Opus 5·xhigh · Sonnet 5·xhigh · Haiku 5.5·medium · Sonnet 5·high · Haiku 5.5·xhigh · Haiku 5.5·low · Sonnet 5·low
IdentityBench v1: identity-governance decisions (680 test cases), Claude CLI baseline prompt
github-identitybench
rlwillen0121/identity-eval-harness README leaderboard; frozen test split, baseline prompt, provider claude_cli; no effort set; score = grounded accuracy %; cost = API-equivalent USD per 1,000 decisions; read 2026-10-15 ✓xmodelSonnet 5.5·medium · Fable 5.1·high · Opus 5.5·medium
IdentityBench v3-Full: 360 multi-turn stateful IGA episodes (max 12 turns), claude_cli provider, 17 Oct 2026
github-identitybench
rlwillen0121/identity-eval-harness README fad1304 22-model v3_full table; experiments/v3-full.yaml; NEW GROUP (held identitybench-v1 is the 680-case single-prompt v1 suite); read 2026-10-18 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Fable 5.1·high
IdentityBench v4.1 v41_lite
github-identitybench
IdentityBench v4.1 v41_lite: 60 enterprise IGA scenarios x 6 governance categories = 360 category tasks, 1,628 required actions, 12-turn cap (T0 single monolith), seed 20261019, 1 run per model, README of 2026-10-20 (commit 56591d0), published 2026-10-19; metric tasks solved of 360; cost = run cost of the whole suite (billed or estimated); read 2026-10-20 ✓xmodelHaiku 5.5·medium · Opus 5.5·medium · Sonnet 5.5·medium
identitybench-v41skill-a1
github-identitybench
—xmodelHaiku 5.5·medium · Opus 5.5·medium · Sonnet 5.5·medium
identitybench-v41skill-a2
github-identitybench
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium
identitybench-v41skill-a3
github-identitybench
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium
identitybench-v41skill-a4
github-identitybench
—xmodelHaiku 5.5·medium
IndustryBench Chinese, Final (SV), non-thinking vs thinking
arxiv-2605.10267
IndustryBench (arXiv 2605.10267 v3): 2,049 Chinese industrial-procurement QA items, zero-shot closed-book, raw 0-3 by Qwen3-Max judge minus safety-violation penalty (Final SV, 0-3); outputs collected Feb 2026; Table 7 = default non-reasoning run, Table 8 = same items/judge with thinking enabled; read 2026-10-19 ✓xmodelSonnet 4.6·nothink
InferenceBench (prompt v1.0.1 (†))
inferencebench.ai
InferenceBench, agents tuning LLM inference serving in Claude Code · geometric-mean speed-up over the PyTorch eager baseline across four scenarios, ±SEM over seed-pair runs in the flags; runs with the benchmark prompt v1.0.1 (†) only; effort from the model name; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max · Fable 5·low
InferenceBench (prompt v1.0)
inferencebench.ai
InferenceBench, agents tuning LLM inference serving in Claude Code · geometric-mean speed-up over the PyTorch eager baseline across four scenarios, ±SEM over seed-pair runs in the flags; runs with the benchmark prompt v1.0 only; effort from the model name; no cost; read 2026-09-30 ✓xmodelOpus 4.8·xhigh · Fable 5·low
IntelligentDocumentProcessing: 250 synthetic German documents, invoice routing correct (single-model systems)
github-idp-invoices
DanielHuette/IntelligentDocumentProcessing README, measured 8 Oct 2026; one shared prompt; no extended thinking for Claude; cost = tokens billed x list price per 1,000 documents; read 2026-10-15 ✓xmodelHaiku 5.5·nothink
ITBench-AA (SRE)
artificialanalysis
Artificial Analysis ITBench-AA (SRE); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 4.7·max
izumo-review-high-detection
qiita-m_izumo
—xmodelOpus 5.5·high · Opus 5.5·xhigh
Japanese Stroke LLM Eval
arxiv-2609.16739
multi-turn Japanese conversation · Fable5/Opus4.7 default, score-only ✓xmodelFable 5·high · Opus 4.7·high
JockeyBench Program track (324 races, daily revision agent)
arxiv-2610.19589
Claude Code with MCP bridge; log gain in nats relative to uniform, mean over runs; no model cost (fees are betting score-yen); read 2026-10-15 ✓xmodelOpus 5·high · Fable 5.1·high · Opus 5.5·high
K3 report Fig.13b: BrowseComp score vs cost per task
moonshot
BrowseComp; K3 with context compaction at 300K tokens; GPT-5.6 Sol per effort medium/high/max (points read from the vector drawing of Fig.13b); cost USD per task as plotted (source of costs not stated); K3 tech report p.32; read 2026-10-19 ✓xmodelSonnet 5·medium · Sonnet 5·high · Sonnet 5·max · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·max
K3 report Table 3: Agent Behavior Bench
moonshot
Agent Behavior Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
K3 report Table 3: Chat All-in-One Bench
moonshot
Chat All-in-One Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
K3 report Table 3: Coding Experience [Claude Code harness]
moonshot
Coding Experience (Moonshot in-house); harness Claude Code; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
K3 report Table 3: DECK Bench
moonshot
DECK Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
K3 report Table 3: Faithfulness
moonshot
Faithfulness (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelOpus 4.8·max
K3 report Table 3: Finance Bench
moonshot
Finance Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelOpus 4.8·max
K3 report Table 3: KWV Bench
moonshot
KWV Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
K3 report Table 3: MIRA Bench
moonshot
MIRA Bench (Moonshot in-house); harness MIRA; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
kamiumi-svg-motion
note-kamiumi_ai
—xmodelSonnet 5.5·medium · Opus 5.5·medium
kenji_harada: 4 hard coding tasks, Opus 5.5 sweep + Fable 5.1 (Claude Code)
qiita-kenji-harada
Claude Code 2.1.283 -p, macOS, 4 hard Node.js tasks x3 runs, median time/cost at API list price; qiita.com/kenji_harada/items/7d8be259208cd3a5843f; read 2026-10-05 ✓xmodelOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Fable 5.1·high
kenji_harada: 6 small coding tasks, Opus 5.5/Opus 5/Fable 5.1 (Claude Code)
qiita-kenji-harada
Claude Code 2.1.283 -p, macOS, 6 small Node.js tasks x3 runs, median time/cost at API list price; qiita.com/kenji_harada/items/7d8be259208cd3a5843f; read 2026-10-05 ✓xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Opus 5·high · Fable 5.1·high
KernelBench CUDA 01 (Claude Code)
github-kernelbench
Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 01, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓xmodelHaiku 5.5·max · Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max · Fable 5.2·high
KernelBench CUDA 02 (Claude Code)
github-kernelbench
Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 02, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓xmodelHaiku 5.5·max · Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max
KernelBench CUDA 03 (Claude Code)
github-kernelbench
Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 03, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓xmodelHaiku 5.5·max · Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max · Fable 5.2·high
KernelBench CUDA 03 (OpenRouter)
github-kernelbench
Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 03, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓xmodelFable 5·high · Opus 5·max
KernelBench CUDA 04 (Claude Code)
github-kernelbench
Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 04, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓xmodelHaiku 5.5·max · Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max
KernelBench CUDA 04 (OpenRouter)
github-kernelbench
Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 04, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓xmodelFable 5·high · Opus 5·max
KernelBench hard 01_fp8_gemm
github-kernelbench
kernelbench.com hard board problem 01_fp8_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓xmodelFable 5·high · Opus 4.8·high · Fable 5·max
KernelBench hard 03_paged_attention
github-kernelbench
kernelbench.com hard board problem 03_paged_attention; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓xmodelFable 5·high · Opus 4.8·high
KernelBench hard 05_topk_bitonic
github-kernelbench
kernelbench.com hard board problem 05_topk_bitonic; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓xmodelFable 5·high · Opus 4.8·high · Opus 5·max
KernelBench hard 07_w4a16_gemm
github-kernelbench
kernelbench.com hard board problem 07_w4a16_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓xmodelFable 5·high · Opus 4.8·high · Opus 5·max · Opus 4.8·max
KernelBench MEGA (Claude Code)
github-kernelbench
02_kimi_linear_decode megakernel on RTX PRO 6000 Blackwell · harness=claude (Claude Code) · n=1 per cell, runs months apart (Jun-Sep 2026); NOT the board's CUDA task (that is the held kernelbench-cc/or group); read 2026-10-01 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Fable 5·high · Fable 5.1·high · Opus 4.8·high
Kimi Code Bench 2.0
moonshot
arXiv 2607.24653 Fig. 13a · Moonshot in-house suite via Claude Code, Opus 4.8 vs Fable 5 at max, measured cost (vector-read) ✓xmodelOpus 4.8·max · Fable 5·max
Kimi K2.7 Code card: Kimi Claw 24/7 Bench
moonshot
Kimi Claw 24/7 Bench; in-house; 17 scenarios/610 evaluation points; OpenClaw harness; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓xmodelOpus 4.8·xhigh
Kimi K2.7 Code card: Kimi Code Bench v2
moonshot
Kimi Code Bench v2; in-house; 10+ languages; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓xmodelOpus 4.8·xhigh
Kimi K2.7 Code card: MCP Atlas
moonshot
MCP Atlas; 100 tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓xmodelOpus 4.8·xhigh
Kimi K2.7 Code card: MCP Mark Verified
moonshot
MCP Mark Verified; human-verified MCPMark; 100-step tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓xmodelOpus 4.8·xhigh
Kimi K2.7 Code card: MLS Bench Lite
moonshot
MLS Bench Lite; 30-task subset; 5 h per task; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓xmodelOpus 4.8·max
Kimi K2.7 Code card: Program Bench
moonshot
Program Bench; 200 tasks; 248k fuzz tests; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓xmodelOpus 4.8·xhigh
Kimi K3 card: AA-Briefcase (Elo)
moonshot
AA-Briefcase Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: AA-LCR
moonshot
AA-LCR; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: Agents' Last Exam
moonshot
Agents' Last Exam; official leaderboard as of 2026-07-23, primary pass-rate metric; per-model harness from the leaderboard; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·xhigh · Opus 4.8·max
Kimi K3 card: APEX-Agents
moonshot
APEX-Agents; Mercor leaderboard as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: AutomationBench
moonshot
AutomationBench; 600-task public subset; official GitHub setup; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: BabyVision w/ python
moonshot
BabyVision with Python; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: CharXiv (RQ) (no tools)
moonshot
CharXiv reasoning questions; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: CharXiv (RQ) (with python)
moonshot
CharXiv reasoning questions; avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: FrontierSWE
moonshot
FrontierSWE; dominance scores recomputed from raw scores, current as of 2026-07-16; K3 Kimi Code, Sol Codex; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: GDPval-AA v2 (Elo)
moonshot
GDPval-AA v2 Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: GPQA Diamond
moonshot
GPQA Diamond; no tools; top-p 0.95; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: HLE-Full (no tools)
moonshot
HLE full set; top-p 0.95; with-tools = general tools; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: HLE-Full (with tools)
moonshot
HLE full set; top-p 0.95; with-tools = general tools; with tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MathVision (no tools)
moonshot
MathVision; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MathVision (with python)
moonshot
MathVision; avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MCP-Atlas
moonshot
MCP-Atlas; 500-task public subset; 100-turn limit; judge Gemini 3.1 Pro; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MCPMark-Verified
moonshot
MCPMark-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MLS-Bench-Lite
moonshot
MLS-Bench-Lite (30-task subset); Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MMMU-Pro (no tools)
moonshot
MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MMMU-Pro (with python)
moonshot
MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: MMVU
moonshot
MMVU; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelOpus 4.8·max
Kimi K3 card: OfficeQA Pro
moonshot
OfficeQA Pro; entire PDF corpus given as images, no machine-readable text; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: OmniDocBench
moonshot
OmniDocBench; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: OSWorld 2.0
moonshot
OSWorld 2.0; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: OSWorld-Verified
moonshot
OSWorld-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: PerceptionBench
moonshot
PerceptionBench (Moonshot in-house benchmark); avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: PostTrainBench
moonshot
PostTrainBench; K3/Fable/Sol official Harbor implementation at max effort, avg of 3 runs on H20 (official: H100); GLM/GPT-5.5/Opus from the official board; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: ProgramBench
moonshot
ProgramBench; K3 with Kimi Code harness; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: ResearchRubrics
moonshot
ResearchRubrics; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelOpus 4.8·max
Kimi K3 card: SaaS-Bench
moonshot
SaaS-Bench; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelOpus 4.8·max
Kimi K3 card: SciCode
moonshot
SciCode; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: SpreadsheetBench 2
moonshot
SpreadsheetBench 2; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: SWE-Marathon
moonshot
SWE-Marathon, H20-calibrated branch (card: official tasks as of 2026-07-09 prior to final v1.1; blog/report: v1.1 tasks); K3/Opus/Fable Claude Code, Sol Codex; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: Toolathlon-Verified
moonshot
Toolathlon-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: Video-MME (w. sub)
moonshot
Video-MME with subtitles; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelOpus 4.8·max
Kimi K3 card: WorldVQA ForceAnswer
moonshot
WorldVQA ForceAnswer variant; vision avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: ZeroBench (pass@5) (no tools)
moonshot
ZeroBench main set; pass@5 (run 5 times, official setting); no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: ZeroBench (pass@5) (with python)
moonshot
ZeroBench main set; pass@5 (run 5 times, official setting); with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
Kimi K3 card: τ³-Banking
moonshot
tau3-Banking; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓xmodelFable 5·max · Opus 4.8·max
kingy930869-4tasks
kingy.ai
—xmodelHaiku 5.5·low · Sonnet 5.5·low · Opus 5.5·low · Fable 5.1·low · Fable 5.2·low
krauq.ai SCAN 2026 CTF qualifier (55 flags, 1 attempt each)
krauq.ai
Claude Code (builds 2.1.273/2.1.284/2.1.293), 55 blockchain-investigation CTF flags, one attempt per flag, 20-min limit; score = flags solved of 55; cost = API-equivalent total of the 55 attempts (sum of per-attempt cost_usd; estimated attempts flagged); read 2026-10-15 ✓xmodelHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5·low · Fable 5·medium · Fable 5.1·low · Fable 5.1·medium
Labelbox · Implicit Intelligence
labelbox.com
Labelbox leaderboards, Implicit Intelligence · score as printed (integer), top five models only; effort from the label; read 2026-09-30 ✓xmodelOpus 4.7·high
latitude easy tier
github-latitude
ai-sdk · 12 single-file bug tasks ×3, 3 Claude models at default effort, list price ✓xmodelOpus 4.8·nothink
latitude-cross6
github-latitude
—xmodelOpus 4.8·nothink
latitude-sweep
github-latitude
—xmodelOpus 5·low · Opus 5·max · Opus 5·medium · Opus 5·high · Opus 5·xhigh
lb2-browser-analysis-driven-action
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-browser-constraint-adherence
livebench
—xmodelFable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-browser-data-aggregation
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-browser-sequential-planning
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-browser-structural-integrity
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-coding-agent-loop
livebench
—xmodelHaiku 5.5·max · Opus 5.5·max
lb2-coding-billing-limits
livebench
—xmodelFable 5.2·xhigh · Haiku 5.5·max · Sonnet 5.5·max
lb2-coding-cloud-infrastructure
livebench
—xmodelFable 5.2·xhigh · Haiku 5.5·max · Sonnet 5.5·max
lb2-coding-hard-fixes
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-coding-llm-api-integration
livebench
—xmodelFable 5.2·xhigh
lb2-coding-product-ux
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-dataanalysis-consecutive-events
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-dataanalysis-table-join
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-dataanalysis-table-reformat
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-design-documents-docx
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-design-pdf
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-design-presentations-ppt
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-longhorizon-delegated-operations
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-longhorizon-finance-compliance-apps
livebench
—xmodelFable 5.1·max · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-longhorizon-long-run-operations
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-longhorizon-operations-commerce-apps
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-longhorizon-performance-optimisation
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-longhorizon-systems-engineering
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-reasoning-logic-with-navigation
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-reasoning-spatial
livebench
—xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-reasoning-theory-of-mind
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-reasoning-zebra-puzzles
livebench
—xmodelHaiku 5.5·max
lb2-research-legal
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-research-regulatory-compliance
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-research-science-evidence
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-research-strategy
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
lb2-research-tax-finance
livebench
—xmodelFable 5.1·max · Fable 5.2·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
leanzero-forge2-medium
leanzero
—xmodelOpus 5.5·medium · Sonnet 5.5·medium
lechmazur Adversarial LLM Debate Benchmark — Bradley-Terry rating
lechmazur-debate
lechmazur/debate, side-swapped adversarial debates, multi-model judge panel (incl. a Claude judge), Bradley-Terry rating with match counts; read 2026-10-05 ✓xmodelFable 5.1·high · Fable 5·high · Opus 5·high · Opus 5.5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high · Sonnet 4.6·high
lechmazur Creative Writing Benchmark — comparison score
lechmazur-writing
lechmazur/writing, matched creative-brief stories judged pairwise by rotating LLM evaluators, Thurstone/BT-style comparison score centered at 0, bootstrap 95% CI; read 2026-10-05 ✓xmodelFable 5.1·high · Opus 5·xhigh · Opus 5.5·high · Opus 5·high · Opus 4.8·xhigh
lechmazur NYT Connections Extended Benchmark — full 940-puzzle board
lechmazur-nyt
lechmazur/nyt-connections, 940 puzzles, quadratic-v1 scoring (g/4)^2 per puzzle, objective (not model-judged); read 2026-10-05 ✓xmodelOpus 5·xhigh · Fable 5·high · Opus 5·high · Opus 4.8·xhigh · Fable 5.1·high · Opus 5.5·high · Opus 4.8·high · Sonnet 4.6·high · Sonnet 5·high · Opus 4.7·high · Sonnet 5.5·high · Haiku 5.5·high
lechmazur NYT Connections Extended Benchmark — harder 100-puzzle subset
lechmazur-nyt
lechmazur/nyt-connections, harder 100-puzzle subset, same quadratic-v1 scoring, objective; read 2026-10-05 ✓xmodelOpus 5·xhigh · Fable 5·high · Opus 4.8·high · Opus 5·high · Opus 4.8·xhigh · Opus 5.5·high · Fable 5.1·high · Sonnet 4.6·high · Sonnet 5·high · Sonnet 5.5·high · Haiku 5.5·high · Opus 4.7·high
legalforecast-mtd-micro-brier
johnhughes3/LegalForecastBench
—xmodelOpus 5.5·high · Sonnet 5.5·high
LIBERO-Agent (30-task embodied-manipulation suite), general-purpose coding agents
arxiv-2609.39507
LIBERO-Agent (arXiv 2609.39507, 30 Sep 2026): 30-task suite, 3 rollouts per task, full observation profile, no demonstrations, 1800 s per rollout; each model in its own agent harness (Table 5); score = Overall Score (Table 1, 0-100); judgement accuracy, short/long-horizon SR and SC are other columns of the same runs; avg time and tokens reported, no dollar cost; read 2026-10-20 ✓xmodelOpus 5·high · Fable 5.1·high
LiveBench 2.0 · Browser Use suite
livebench
LiveBench 2.0 Browser Use suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.1·max
LiveBench 2.0 · Coding suite
livebench
LiveBench 2.0 Coding suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.1·max
LiveBench 2.0 · Data Analysis suite
livebench
LiveBench 2.0 Data Analysis suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.1·max
LiveBench 2.0 · Design suite
livebench
LiveBench 2.0 Design suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.1·max
LiveBench 2.0 · Long-Horizon suite
livebench
LiveBench 2.0 Long-Horizon suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.1·max
LiveBench 2.0 · Reasoning suite
livebench
LiveBench 2.0 Reasoning suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.1·max
LiveBench 2.0 · Research suite
livebench
LiveBench 2.0 Research suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelFable 5·max · Opus 4.8·max · Opus 5·max · Sonnet 5·max · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.1·max
livebench-task-amps_hard
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-code_completion
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-code_generation
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-connections
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-consecutive_events
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-integrals_with_game
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-javascript
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-logic_with_navigation
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-math_comp
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-olympiad
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-paraphrase
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-plot_unscrambling
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-python
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-simplify
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-spatial
livebench
—xmodelSonnet 4.6·low · Opus 4.7·low · Opus 4.7·medium
livebench-task-story_generation
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-summarize
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-tablejoin
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-tablereformat
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-theory_of_mind
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-typescript
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-typos
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.7·medium
livebench-task-zebra_puzzle
livebench
—xmodelSonnet 4.6·high · Sonnet 4.6·low · Opus 4.7·low · Opus 4.7·medium
llm_chess — aggregated Elo vs Random + Dragon engine (all Claude models)
llm_chess
maxim-saplin/llm_chess, LLM plays Black vs RANDOM_PLAYER and Dragon engine, Elo anchored to Dragon ladder, 95% CI, average_game_cost as cost per game; read 2026-10-05 ✓xmodelOpus 5·high · Sonnet 5·high · Opus 5.5·high · Sonnet 4.6·high · Opus 4.8·high · Haiku 4.5·nothink · Opus 4.7·high · Sonnet 5.5·high · Haiku 5.5·high
llm_chess — single 10-game match vs RANDOM_PLAYER (Black win rate)
llm_chess
maxim-saplin/llm_chess raw per-match _aggregate_results.json, n=10 games per model, win% computed by us from black_wins/total_games; read 2026-10-05 ✓xmodelSonnet 5.5·high
llm-regression product suite 2026-10-03 (quality /10)
nikoamoretti-llm-regression
nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-03, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓xmodelOpus 5.5·high
llm-regression product suite 2026-10-04 (quality /10)
nikoamoretti-llm-regression
nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-04, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓xmodelOpus 5.5·high
llm-regression product suite 2026-10-05 (quality /10)
nikoamoretti-llm-regression
nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-05, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓xmodelOpus 5.5·high
llm-regression product suite 2026-10-06 (quality /10)
nikoamoretti-llm-regression
nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-06, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓xmodelOpus 5.5·high
llm-regression product suite 2026-10-07 (quality /10)
nikoamoretti-llm-regression
nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-07, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓xmodelOpus 5.5·high
llm2014-logic-2026-10-median
llm2014-logic
—xmodelSonnet 5.5·xhigh · Haiku 5.5·xhigh
LLMConfBench
arxiv-2609.20666
arXiv 2609.20666 · conformer energy ranking, 27 molecules × 3 prompts via OpenRouter, Opus 5 and Sonnet 5 at high; cost summed from the repo's per-call logs ✓xmodelOpus 5·high · Sonnet 5·high
llmreg-product-pass-2026-10-03
nikoamoretti-llm-regression
—xmodelOpus 5.5·high
llmreg-product-pass-2026-10-04
nikoamoretti-llm-regression
—xmodelOpus 5.5·high
llmreg-product-pass-2026-10-05
nikoamoretti-llm-regression
—xmodelOpus 5.5·high
llmreg-product-pass-2026-10-06
nikoamoretti-llm-regression
—xmodelOpus 5.5·high
llmreg-product-pass-2026-10-07
nikoamoretti-llm-regression
—xmodelOpus 5.5·high
LMArena Agent
lmarena.ai
Agent Mode real sessions (Code/Chat/Work) · cost=median $/task (14d window), score=Net Improvement composite signal, effort read from model-variant label; live board, read 27 Sep 2026 ✓xmodelFable 5.2·max · Opus 5.5·max · Opus 5.5·high · Sonnet 5.5·max · Sonnet 5.5·high · Fable 5.1·max · Fable 5·high · Opus 5·max · Sonnet 5·high · Haiku 5.5·high
LMArena Document
lmarena.ai
Document arena (Overall, the page's default board: no style control) · blind pairwise votes on answers about uploaded documents, Elo rating, no cost; vote cutoff 13 Sep 2026, so no 5.5 model yet; effort read from the arena's model key (-thinking, -high, -max); live board, read 30 Sep 2026 ✓xmodelOpus 4.8·nothink · Fable 5.1·max · Opus 5·high · Fable 5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high
LMArena Search
lmarena.ai
Search arena (Overall, the page's default board: no style control) · blind pairwise votes on answers with web search, Elo rating, no cost; each model runs with the arena's search tool (key suffix -search); vote cutoff 24 Aug 2026, no 5.5 model; effort not in the key (default), Fable 5 at high per its display name; live board, read 30 Sep 2026 ✓xmodelOpus 4.8·nothink · Opus 5·high · Fable 5.1·high · Fable 5·high · Opus 5.5·medium
LMArena Text
lmarena.ai
Text arena (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-opus-5.5-high-vertex); live board, read 30 Sep 2026 ✓xmodelOpus 4.8·nothink · Opus 5.5·high · Sonnet 5.5·xhigh · Fable 5.1·max · Opus 5·max · Opus 5·high · Fable 5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high · Haiku 5.5·high
LMArena Vision
lmarena.ai
Vision arena (Overall, the page's default board: style control) · blind pairwise votes on image-input prompts, Elo rating, no cost; vote cutoff 28 Sep 2026, no 5.5 model yet; effort read from the arena's model key (-thinking, -thinking-16k/32k, -high, -max); live board, read 30 Sep 2026 ✓xmodelOpus 4.8·nothink · Fable 5·high · Opus 5.5·high · Fable 5.1·max · Opus 5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5.5·xhigh · Sonnet 5·high
LMArena WebDev
lmarena.ai
Code arena, WebDev board (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-sonnet-5-5-high); live board, read 30 Sep 2026 ✓xmodelOpus 4.8·nothink · Fable 5.2·xhigh · Opus 5.5·max · Sonnet 5.5·xhigh · Fable 5.1·max · Sonnet 5.5·high · Opus 5·max · Opus 5·high · Fable 5·high · Haiku 5.5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high
LMArena WebDev — HTML sub-category
lmarena.ai
lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-html, latest split, publish 2026-10-02; read 2026-10-05 ✓xmodelOpus 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·high · Fable 5.1·max · Opus 5·max · Fable 5·high · Opus 5·high · Haiku 5.5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high · Opus 4.8·nothink
LMArena WebDev — React sub-category
lmarena.ai
lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-react, latest split, publish 2026-10-02; read 2026-10-05 ✓xmodelOpus 5.5·max · Sonnet 5.5·xhigh · Fable 5.1·max · Sonnet 5.5·high · Opus 5·max · Opus 5·high · Fable 5·high · Haiku 5.5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high · Opus 4.8·nothink
lob-code-fix-merge-intervals
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-code-lru-cache
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-code-parse-duration
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-hard-coin-faces
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-hard-count-multiples
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-hard-lex-topo-order
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-hard-locking-tradeoff
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-hard-parse-csv-line
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-hard-postmortem-rewrite
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-hard-puzzle-hunt
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-hard-weighted-intervals
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh
lob-reason-race-order
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-reason-snail-pole
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-reason-trailing-zeros
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-write-container-summary
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-write-https-explainer
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
lob-write-launch-email
github-low-or-bust
—xmodelFable 5·high · Fable 5·low · Fable 5·medium · Fable 5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·xhigh
M-GATE grammar error detection, MCC mean over 30 languages (temperature 0)
mgate-2608.03803
M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.1; 53 models / 82 configurations March-July 2026; linguist-crafted adversarial grammar items, 30 languages, temperature 0, one run; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓xmodelOpus 4.8·nothink · Sonnet 5·high
M-GATE round-trip translation, hard subset, normalised score mean over 29 targets
mgate-2608.03803
M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.2; 53 models / 82 configurations March-July 2026; 100 English sources round-tripped into 29 languages, hard subset of 50 sentences, judges GPT-5.4 Mini + Claude Haiku 4.5 + Gemini 3.1 Flash Lite at temp 0; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓xmodelOpus 4.8·nothink · Sonnet 5·high
MA-Bench mission analysis, benchmark condition, weighted score (GPT-5.6 judge), Opus 5 / Fable 5.1 / Sonnet 5
arxiv-2610.19138
arXiv 2610.19138 Table 4; production judge GPT-5.6; 3000-word cap; 216 briefs (n=30/50/30 per model); read 2026-10-18 ✓xmodelOpus 5·medium · Fable 5.1·medium · Sonnet 5·medium
MathArena · aime 2026
matharena.ai
MathArena, aime--aime_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
MathArena · apex 2025
matharena.ai
MathArena, apex--apex_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
MathArena · april
matharena.ai
MathArena, arxiv--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelFable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·xhigh
MathArena · april
matharena.ai
MathArena, arxiv_false--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelFable 5·max · Opus 4.8·max · Opus 4.7·xhigh
MathArena · august
matharena.ai
MathArena, arxiv--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelFable 5.1·high · Opus 5.5·high · Fable 5.1·low
MathArena · august
matharena.ai
MathArena, arxiv_false--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelFable 5.1·high · Fable 5.1·low · Opus 5.5·high
MathArena · february
matharena.ai
MathArena, arxiv--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
MathArena · february
matharena.ai
MathArena, arxiv_false--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
MathArena · hmmt feb 2026
matharena.ai
MathArena, hmmt--hmmt_feb_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
MathArena · january
matharena.ai
MathArena, arxiv--january: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.7·xhigh
MathArena · june
matharena.ai
MathArena, arxiv--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelFable 5.1·max · Fable 5·max · Opus 5.5·high · Opus 5·max · Opus 4.8·max · Fable 5.1·low
MathArena · june
matharena.ai
MathArena, arxiv_false--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 5·max · Fable 5.1·low · Fable 5·max · Opus 4.8·max
MathArena · june
matharena.ai
MathArena, arxivlean--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 5·max · Opus 4.8·max · Fable 5·max
MathArena · kangaroo 2025 1-2
matharena.ai
MathArena, kangaroo--kangaroo_2025_1-2: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max
MathArena · kangaroo 2025 11-12
matharena.ai
MathArena, kangaroo--kangaroo_2025_11-12: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max
MathArena · kangaroo 2025 3-4
matharena.ai
MathArena, kangaroo--kangaroo_2025_3-4: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max
MathArena · kangaroo 2025 5-6
matharena.ai
MathArena, kangaroo--kangaroo_2025_5-6: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max
MathArena · kangaroo 2025 7-8
matharena.ai
MathArena, kangaroo--kangaroo_2025_7-8: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max
MathArena · kangaroo 2025 9-10
matharena.ai
MathArena, kangaroo--kangaroo_2025_9-10: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max
MathArena · march
matharena.ai
MathArena, arxiv--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
MathArena · march
matharena.ai
MathArena, arxiv_false--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
MathArena · may
matharena.ai
MathArena, arxiv--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelFable 5·max · Fable 5.1·max · Opus 5.5·high · Opus 4.8·max · Fable 5.1·low
MathArena · may
matharena.ai
MathArena, arxiv_false--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 5.5·high · Fable 5.1·low · Fable 5·max · Opus 4.8·max
MathArena · shortlist 2025
matharena.ai
MathArena, apex--shortlist_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelOpus 4.8·max · Opus 4.7·xhigh
matharena-arxiv-false-october
matharena.ai
—xmodelOpus 5.5·high
matharena-arxiv-false-september
matharena.ai
—xmodelOpus 5.5·high
matharena-arxiv-october
matharena.ai
—xmodelOpus 5.5·high
matharena-arxiv-september
matharena.ai
—xmodelOpus 5.5·high
matharena-arxivlean-september
matharena.ai
—xmodelOpus 5.5·high
matharena-leanlean-oct26
matharena.ai
—xmodelOpus 5.5·high
MazeBench ASCII no tools
mazebench
MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, no Python tools; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓xmodelOpus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max
MazeBench ASCII with Python tools
mazebench
MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, Python tools enabled; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓xmodelOpus 5·max · Fable 5·max · Fable 5.1·max · Haiku 5.5·max · Sonnet 5·max
MCP-Atlas public set (GLM-5.2 card)
zhipu
MCP-Atlas 500-task public set, think mode, 10 min timeout, judge Gemini-3.0-Pro; table value 76.8 (chart shows 77.0); read 2026-10-19 ✓xmodelOpus 4.8·max
mcptox
github-mcptox
—xmodelOpus 5·high · Opus 4.8·nothink · Sonnet 5·high
MedChem (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelSonnet 5·max · Fable 5.1·max
MediCrab Lab notes #2: 24 planted errors in one pediatric septic-shock trial
medicrab
same prompt, no web search, 2-3 runs each, graded blind; score = mean errors found of 24; read 2026-10-15 ✓xmodelOpus 5.5·medium · Fable 5.1·medium · Sonnet 5.5·medium · Haiku 5.5·high
Melvynx medium duo
github-melvynx
Claude Code · 2 apps, Fable 5 vs Sonnet 5 medium, cost only ✓xmodelFable 5·medium · Sonnet 5·medium
Melvynx xhigh trio
github-melvynx
Claude Code · 3 apps, Fable 5 vs Sonnet 5 xhigh, same batch, verified runs, cost only ✓xmodelFable 5·xhigh · Sonnet 5·xhigh
Melvynx/benchmarks: web-app build, Claude Code, Opus 5.5 vs Sonnet 5.5 medium, 2 tasks without cursor continuation
github-melvynx
Melvynx/benchmarks runs 2026-09-28; harness cc (Claude Code); tasks timezone-checker + video-generation summed; cost = accounting.json costUsd (harness cost, list basis); no quality score in repo (harness-verified only); read 2026-10-18 ✓xmodelSonnet 5.5·medium
merc-bench hard tier T13 (per-task, adopted as core at 5e09655)
github-merc-bench
github.com/sirneb/merc-bench, task T13, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 5.5·low · Haiku 5.5·medium · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·xhigh
merc-bench hard tier T14 (per-task, adopted as core at 5e09655)
github-merc-bench
github.com/sirneb/merc-bench, task T14, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·xhigh
merc-bench hard tier T15 (per-task, adopted as core at 5e09655)
github-merc-bench
github.com/sirneb/merc-bench, task T15, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 5.5·low · Haiku 5.5·medium · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·xhigh
Mercor · AdvancedIF
mercor.com
Mercor, AdvancedIF: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelFable 5.1·max · Opus 5.5·max · Fable 5.1·high
Mercor · BrowseComp Extended
mercor.com
Mercor, BrowseComp Extended: the public benchmark extended with new expert-built Mercor tasks (100 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelSonnet 5.5·max · Fable 5.1·max · Opus 5.5·max
Mercor · CharXiv
mercor.com
Mercor, CharXiv: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Sonnet 5.5·max
Mercor · CharXiv Extended
mercor.com
Mercor, CharXiv Extended: the public benchmark extended with new expert-built Mercor tasks (150 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Sonnet 5.5·max
Mercor · CVDP
mercor.com
Mercor, CVDP: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5.5·max
Mercor · DeepResearch Bench II
mercor.com
Mercor, DeepResearch Bench II: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5·high · Fable 5.1·max · Sonnet 4.6·high · Opus 5.5·max
Mercor · DeepSWE v1.1
mercor.com
Mercor, DeepSWE v1.1: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Opus 5·max
Mercor · FrontierSWE v2
mercor.com
Mercor, FrontierSWE v2: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelFable 5.1·high · Opus 5·high
Mercor · GDPval
mercor.com
Mercor, GDPval: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max · Sonnet 5.5·max · Opus 5·max
Mercor · GDPval Extended
mercor.com
Mercor, GDPval Extended: the public benchmark extended with new expert-built Mercor tasks (132 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Haiku 5.5·max · Fable 5·max · Opus 5·max
Mercor · Harvey LAB
mercor.com
Mercor, Harvey LAB: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5·max · Fable 5.1·max · Fable 5·max
Mercor · HLE
mercor.com
Mercor, HLE: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·high
Mercor · HLE Extended
mercor.com
Mercor, HLE Extended: the public benchmark extended with new expert-built Mercor tasks (447 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5·high · Fable 5·high
Mercor · Long Context Reasoning (AA-LCR v1.1)
mercor.com
Mercor, Long Context Reasoning (AA-LCR v1.1): the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelFable 5.1·max · Fable 5·max · Opus 5.5·max
Mercor · Long Context Reasoning (AA-LCR) Extended
mercor.com
Mercor, Long Context Reasoning (AA-LCR) Extended: the public benchmark extended with new expert-built Mercor tasks (89 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·high
Mercor · MedXpertQA MM
mercor.com
Mercor, MedXpertQA MM: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max
Mercor · MedXpertQA MM Extended
mercor.com
Mercor, MedXpertQA MM Extended: the public benchmark extended with new expert-built Mercor tasks (51 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·high
Mercor · MMLU-Pro
mercor.com
Mercor, MMLU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5.1·high
Mercor · MMMU-Pro
mercor.com
Mercor, MMMU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max · Sonnet 5.5·max
Mercor · MMMU-Pro Extended
mercor.com
Mercor, MMMU-Pro Extended: the public benchmark extended with new expert-built Mercor tasks (440 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max
Mercor · ProgramBench
mercor.com
Mercor, ProgramBench: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·high · Opus 5·high
Mercor · SciCode
mercor.com
Mercor, SciCode: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Fable 5.1·max · Fable 5·max · Sonnet 5.5·max
Mercor · SUPERChem
mercor.com
Mercor, SUPERChem: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Opus 5·max · Fable 5.1·max
Mercor · SWE Atlas - Codebase QnA
mercor.com
Mercor, SWE Atlas - Codebase QnA: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5·max · Sonnet 5.5·max · Opus 5.5·max
Mercor · SWE Atlas - Codebase QnA Extended
mercor.com
Mercor, SWE Atlas - Codebase QnA Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelSonnet 5.5·max · Opus 5.5·max · Opus 5·max · Fable 5.1·max
Mercor · SWE-bench Multilingual
mercor.com
Mercor, SWE-bench Multilingual: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5·max · Opus 5.5·max
Mercor · SWE-bench Verified
mercor.com
Mercor, SWE-bench Verified: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max
Mercor · SWE-bench Verified Extended
mercor.com
Mercor, SWE-bench Verified Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5·max · Fable 5·max
Mercor · Terminal-Bench 2.1 Extended
mercor.com
Mercor, Terminal-Bench 2.1 Extended: the public benchmark extended with new expert-built Mercor tasks (99 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 5.5·max · Sonnet 5.5·max
Mercor APEX-Accounting
mercor.com
Mercor's own board, APEX-Accounting, harness loop_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelFable 5.2·max · Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Opus 5·max · Haiku 5.5·max · Fable 5·max · Opus 4.7·max · Sonnet 5·max · Opus 4.8·max · Opus 5.5·medium
Mercor APEX-Agents
mercor.com
Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelFable 5.2·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max · Fable 5.1·high · Sonnet 4.6·high · Opus 5.5·medium · Opus 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·max · Haiku 5.5·max
Mercor APEX-SWE (inspect)
mercor.com
Mercor's own board, APEX-SWE, harness inspect · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelOpus 4.8·high · Sonnet 5·high
Mercor APEX-v1 (The AI Productivity Index)
mercor.com
Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelSonnet 4.6·high · Fable 5·max · Opus 4.7·max · Sonnet 5·max · Opus 4.8·max · Opus 5·xhigh · Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max
Mercor open-source · GeneBench-Pro
mercor.com
Mercor open-source evaluation, effort max (board field); read 2026-10-15 ✓xmodelSonnet 5.5·max · Opus 5.5·max
Mercor open-source · Terminal-Bench 4.0
mercor.com
Mercor open-source evaluation, effort max (board field); read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Sonnet 5.5·max
mercor-apexacct-mean
mercor.com
—xmodelFable 5.2·max · Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Opus 5·max · Haiku 5.5·max · Fable 5·max · Opus 4.7·max · Sonnet 5·max · Opus 4.8·max · Opus 5.5·medium
mercor-apexv1-mean
mercor.com
—xmodelSonnet 4.6·high · Fable 5·max · Opus 4.7·max · Sonnet 5·max · Opus 4.8·max · Opus 5·xhigh · Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max
messydocs-docexact
github-messydocsbench
—xmodelOpus 5.5·medium · Sonnet 5·high
messydocs-docexact-contracts
github-messydocsbench
—xmodelOpus 5.5·medium · Sonnet 5·high
messydocs-docexact-invoices
github-messydocsbench
—xmodelOpus 5.5·medium · Sonnet 5·high
messydocs-docexact-receipts
github-messydocsbench
—xmodelOpus 5.5·medium · Sonnet 5·high
messydocs-docexact-sroie
github-messydocsbench
—xmodelOpus 5.5·medium · Sonnet 5·high
messydocs-docexact-statements
github-messydocsbench
—xmodelSonnet 5·high
messydocs-docexact-w2
github-messydocsbench
—xmodelOpus 5.5·medium · Sonnet 5·high
Mideind Icelandic LLM leaderboard, Case Government (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_case_marking@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Completion (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_cloze@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Famous Phrases (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_phrases@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Flora & Fauna (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_nature_translations@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Generative Quality (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark generative_quality@v1 primary metric quality_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Icelandic GEC (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_gec@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Icelandic WikiQA (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_wikiqa@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Meanings (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_meanings@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Song Lyrics (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_song_lyrics@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mideind Icelandic LLM leaderboard, Textile Terms (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_textiles@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Sonnet 5·high · Haiku 5.5·high · Haiku 4.5·nothink
Mine Odyssey: Minecraft spatial navigation (success rate, 180 tasks)
arxiv-2610.11328
180 navigation tasks (125 outdoor/55 indoor), 500 decisions or 6 h; Opus 5 at 4 efforts plus Opus 5.5 max; Table 1 of arXiv 2610.11328; read 2026-10-15 ✓xmodelOpus 5.5·max · Opus 5·max · Opus 5·high · Opus 5·medium · Opus 5·low
MineBench voxel arena
minebench
minebench.ai live API · voxel builds, 15 prompts, Elo arena read 29 Sep, measured cost for 8 of 12 Claude models ✓xmodelOpus 5.5·max · Opus 5·max · Sonnet 5.5·max · Fable 5.1·max
minilab: self-repo code review, Sonnet 5.5 (medium/high) vs Opus 5.5 (medium)
note-minilab
note.com/minilab/n/n7ccd7f6a6421; author's own notectl CLI repo, 2 questions, 1 run per setting, Claude Code; cost = /usage session total (Q1+Q2) at list price, approximate (article uses 約); effort-from-config: "Sonnet 5.5のmedium、Sonnet 5.5のhigh、Opus 5.5。Opusのeffortは確認して、mediumでした"; read 2026-10-01 ✓xmodelSonnet 5.5·medium · Sonnet 5.5·high
model-routing-eval: 3 fromager ticket tasks (r01t-r03t)
github-model-routing-eval
andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['r01t_fromager_distinfo_ticket', 'r02t_fromager_age_filter_ticket', 'r03t_fromager_dep_chain_ticket']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓xmodelHaiku 5.5·low · Opus 5.5·medium · Sonnet 5.5·medium
model-routing-eval: 8 small coding tasks (t04-t12 subset)
github-model-routing-eval
andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['t04_lru_cache', 't05_cli_json_flag', 't06_log_parser', 't07_mutable_default', 't08_timezone_bug', 't09_thread_safety', 't11_async_migration', 't12_needle_bug']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓xmodelHaiku 5.5·low · Sonnet 5·medium
moraru-opus5-aa-coding-agent-index
gabrielmoraru.com
—xmodelOpus 5·high · Opus 5·xhigh · Opus 5·max
moraru-opus5-cursorbench
gabrielmoraru.com
—xmodelOpus 5·high · Opus 5·xhigh · Opus 5·max
moraru-opus5-frontier-bench-v01
gabrielmoraru.com
—xmodelOpus 5·high · Opus 5·xhigh · Opus 5·max
morinokenjin 6th comparison: lesson slides (photosynthesis; template)
zenn-morinokenjin
claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
morinokenjin 6th comparison: presentation slides (new service)
zenn-morinokenjin
claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
morinokenjin 6th comparison: report slides (library usage)
zenn-morinokenjin
claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender animation judged by Gemini 3.8 Flash (n=1)
zenn-morinokenjin
Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender mechanical watch judged by Gemini 3.8 Flash (n=1)
zenn-morinokenjin
Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender still life judged by Gemini 3.8 Flash (n=1)
zenn-morinokenjin
Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
Morphology (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelSonnet 5.5·max · Fable 5.1·max · Opus 5.5·max
mpgone-5-task-battery
mpgone
—xmodelFable 5.2·medium · Fable 5.2·high · Fable 5.2·max · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·max · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·max
mpgone-benchone-v2-2
mpgone
—xmodelOpus 5.5·medium · Fable 5.1·medium · Fable 5.2·medium · Sonnet 5.5·medium · Haiku 5.5·medium · Sonnet 5·medium · Opus 5·medium
mpgone-benchone-v2-2-coding
mpgone
—xmodelSonnet 5.5·medium · Haiku 5.5·medium · Sonnet 5·medium · Opus 5·medium
mpgone-benchone-v2-2-data-math
mpgone
—xmodelSonnet 5.5·medium · Haiku 5.5·medium · Sonnet 5·medium · Opus 5·medium
mpgone-benchone-v2-2-facts-honesty
mpgone
—xmodelOpus 5.5·medium · Fable 5.1·medium · Fable 5.2·medium · Sonnet 5.5·medium · Haiku 5.5·medium · Sonnet 5·medium · Opus 5·medium
mpgone-benchone-v2-2-long-context
mpgone
—xmodelSonnet 5.5·medium · Haiku 5.5·medium · Sonnet 5·medium
mpgone-benchone-v3-1
mpgone
—xmodelOpus 5.5·medium · Fable 5.2·medium · Sonnet 5.5·medium · Opus 5·medium · Fable 5.1·medium · Haiku 5.5·medium · Sonnet 5·medium
mpgone-benchone-v3-1-agents
mpgone
—xmodelFable 5.1·medium · Haiku 5.5·medium · Sonnet 5·medium
mpgone-benchone-v3-1-coding
mpgone
—xmodelFable 5.1·medium · Haiku 5.5·medium · Sonnet 5·medium
mpgone-benchone-v3-1-data-math
mpgone
—xmodelSonnet 5.5·medium · Haiku 5.5·medium
mpgone-benchone-v3-1-instructions-writing
mpgone
—xmodelOpus 5.5·medium · Fable 5.2·medium · Sonnet 5.5·medium · Opus 5·medium · Haiku 5.5·medium · Sonnet 5·medium
mpgone-benchone-v3-1-long-context
mpgone
—xmodelOpus 5.5·medium · Sonnet 5·medium
mpgone-benchone-v3-1-reasoning
mpgone
—xmodelOpus 5.5·medium · Fable 5.2·medium · Sonnet 5.5·medium · Opus 5·medium · Fable 5.1·medium · Haiku 5.5·medium · Sonnet 5·medium
mre-uxd-u01
github-model-routing-eval
—xmodelHaiku 5.5·low · Haiku 5.5·medium · Opus 5.5·low · Opus 5.5·medium · Sonnet 5.5·low · Sonnet 5.5·medium
mre-uxd-u01-pass
github-model-routing-eval
—xmodelHaiku 5.5·low · Haiku 5.5·medium · Opus 5.5·low · Opus 5.5·medium · Sonnet 5.5·low · Sonnet 5.5·medium
mre-uxdws-u02
github-model-routing-eval
—xmodelHaiku 5.5·low · Haiku 5.5·medium · Opus 5.5·low · Opus 5.5·medium · Sonnet 5.5·low · Sonnet 5.5·medium
NeutronGym McStasBench seen tasks (of 16), one-shot
arxiv-2610.03631
16 seen McStasBench tasks, one episode each, temperature 0; Table 1 of arXiv 2610.03631; read 2026-10-15 ✓xmodelHaiku 4.5·nothink
Next.js Agent Evals (Vercel), base run
nextjs-evals
31 Next.js agent evals, 4 runs each (earlyExit), Claude Code via AI Gateway, no AGENTS.md; https://nextjs.org/evals, github.com/vercel/next-evals-oss@38cb8ad, agent-results.json metadata.experiments[].baseSuccessRate/avgCostUsd; read 2026-10-01 ✓xmodelFable 5·high · Fable 5.1·high · Opus 5.5·high
Next.js Agent Evals (Vercel), with AGENTS.md
nextjs-evals
Same 31 evals/4 runs/Claude Code setup as nextjsevals-base, but with an AGENTS.md/CLAUDE.md file telling the agent the Next.js version has breaking changes; score = docsSuccessRate (board); cost = mean of per-eval costUsd in agent-results.json['results'][model] (computed-by-us, since the board's avgCostUsd field is the base-run figure, not the AGENTS.md-run figure — the per-eval array's own mean, 0.2456 for Opus 5.5, differs from the base 0.234, confirming the two are different runs); read 2026-10-01 ✓xmodelFable 5·high · Fable 5.1·high · Opus 5.5·high
NL2Repo (GLM-5.2 card)
zhipu
NL2Repo temperature=1.0 top_p=1.0 max_new_tokens=48k 400k context, rule+LLM anti-hack; GLM-5.2 card; read 2026-10-19 ✓xmodelOpus 4.8·max
nnakapa-o47o48-t2
zenn-nnakapa
—xmodelOpus 4.7·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh
nnakapa-o5s5-t2
zenn-nnakapa
—xmodelSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh
nnakapa-s5cc-t2
zenn-nnakapa
—xmodelSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
nnakapa-s5cop-t2
zenn-nnakapa
—xmodelSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh
NonoBench
nonobench.com
NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓xmodelSonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
note AI実験室 #1 arithmetic_parser (165 hidden pytest tests)
note-sane_mink364
Claude Code CLI (subscription); hidden pytest; API-equivalent cost from tokens x list price; Haiku 5.5 low/medium vs Opus 5.5 medium; read 2026-10-15 ✓xmodelHaiku 5.5·medium · Opus 5.5·medium
note AI実験室 #1 duration_parse (29 hidden pytest tests)
note-sane_mink364
Claude Code CLI (subscription); hidden pytest; API-equivalent cost from tokens x list price; Haiku 5.5 low/medium vs Opus 5.5 medium; read 2026-10-15 ✓xmodelHaiku 5.5·medium · Opus 5.5·medium
note AI実験室 #2-#4 spec contradiction detection, hard (28 req x 3 specs x 5 runs)
note-sane_mink364
Claude Code CLI; 3 specs x 5 runs = 15 per condition; F1 on contradiction IDs (auto-scored; code fences stripped for all); cost API-equivalent from tokens x price; read 2026-10-15 ✓xmodelHaiku 5.5·medium · Haiku 5.5·max · Opus 4.8·high
nurb-bit-block
github-nurb
—xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh
nurb-bundle-holder
github-nurb
—xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5·high · Sonnet 5·low
nurb-cable-clip
github-nurb
—xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·xhigh · Sonnet 5·medium
nurb-leg-cup
github-nurb
—xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Opus 5·high · Opus 5·low · Opus 5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh · Opus 5·medium · Opus 5·max
nurb-pole-rest
github-nurb
—xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·high · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5·low
nurb-valve-knob
github-nurb
—xmodelFable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Opus 5·high · Opus 5·low · Sonnet 5·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh · Opus 5·xhigh · Opus 5·max · Opus 5·medium
Nuxt agent evals (nuxt/nuxt-evals), Claude Code via AI Gateway
nuxt/nuxt-evals
agent-results.json exportedAt 2026-09-22; 31 Nuxt agent evals, runs: 4, earlyExit: true; score is avgPassRate (fraction of the 4 runs passing, averaged over 31 evals), *100 for a percentage; cost is the gateway's avg cost USD per eval run; models ran on different dates (opus-5.5 2026-09-22, the rest 2026-08-27 to 09-05); read 2026-10-01 ✓xmodelFable 5·high · Fable 5.1·high · Opus 4.8·high · Opus 5·high · Opus 5.5·medium · Sonnet 5·high · Sonnet 5.5·medium
nuxt-evals-success
nuxt/nuxt-evals
—xmodelFable 5·high · Fable 5.1·high · Opus 4.8·high · Opus 5·high · Opus 5.5·medium · Sonnet 5·high · Sonnet 5.5·medium
NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections)
matsonj/eval-connections
results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓xmodelOpus 5.5·low · Fable 5·low · Fable 5.1·low · Sonnet 5.5·low · Opus 5·low · Sonnet 5·low · Haiku 5.5·low
oai-t61-agents-last-exam-v1
openai
—xmodelOpus 5.5·max
oai-t61-osworld-2-1-offline
openai
—xmodelSonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
oai-t61-terminal-bench-4-0
openai
—xmodelSonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
ockbench-top200
ockbench
—xmodelOpus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5·low · Opus 4.8·xhigh · Opus 5·medium · Opus 4.8·medium · Opus 4.8·high · Opus 4.7·xhigh · Opus 4.8·max · Opus 4.8·low
OfficeQA (S55)
anthropic
Sonnet 5.5 card p132 · 3 models at max, score-only ✓xmodelSonnet 5.5·max · Opus 5.5·max · Sonnet 5·max
OfficeQA Pro (S55)
anthropic
Sonnet 5.5 card p132 · 133 q, 3 models at max, score-only ✓xmodelSonnet 5.5·max · Opus 5.5·max · Sonnet 5·max
OpenAgents briefing-model factorial (control)
github-openagents
Claude Code (coder-connect agent) · frozen historical reserve issue, control arm, Opus 5.5 vs Sonnet 5.5 × medium effort, n=4 reps each; score=pass-at-first/final-attempt%, cost=cumulative list-price USD; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
OpenAgents briefing-model factorial (treatment)
github-openagents
Claude Code (coder-connect agent) · frozen historical reserve issue, treatment arm, Opus 5.5 vs Sonnet 5.5 × medium effort, n=4 reps each; score=pass-at-first/final-attempt%, cost=cumulative list-price USD; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
openclaw-opus55-goal-build
openclawdatabase
—xmodelOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
OpenMC neutron-shielding single-shot script generation (v2): execution gate, L1-L5
zenodo-neutron-shielding
zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) Table §5.1 'the gate' (scripts that ran, of 15 total, L1-L5); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
OpenMC neutron-shielding single-shot script generation (v2): physics-correct, L2-L4
zenodo-neutron-shielding
zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) §5.2 (physics-correct of 9, L2-L4, the rungs with a scorable physics criterion); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium
OpenRUA: Claude Code driving a robot, LIBERO-10 (LIBERO-PRO)
arxiv-2610.02459
200 trials per configuration across both perturbation cells, Claude Code at high effort; Table 11 of arXiv 2610.02459; read 2026-10-15 ✓xmodelSonnet 5·high · Opus 5·high · Fable 5.1·high
OS-Bench cross-platform migration, ClaudeCode, Opus 4.8 vs Sonnet 5
arxiv-2610.19368
arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 91 Linux tasks; medium thinking effort; read 2026-10-18 ✓xmodelOpus 4.8·medium · Sonnet 5·medium
OS-Bench cross-platform migration, SWE-agent, Opus 4.8 vs Sonnet 5
arxiv-2610.19368
arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 91 Linux tasks; medium thinking effort; read 2026-10-18 ✓xmodelOpus 4.8·medium · Sonnet 5·medium
OS-Bench cross-platform migration, Win-agent, Opus 4.8 vs Sonnet 5
arxiv-2610.19368
arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 49 Windows tasks; medium thinking effort; read 2026-10-18 ✓xmodelOpus 4.8·medium
OSWorld 2.0 batch
arxiv-2606.29537
arXiv 2606.29537 · 108 workflows, 500 steps, batched tool calls, max ✓xmodelOpus 4.8·max · Opus 4.7·max
OSWorld 2.0 code_mode=false (OpenAI GPT-5.6 post chart)
openai
OSWorld 2.0, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓xmodelOpus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max
OSWorld 2.0 single
arxiv-2606.29537
arXiv 2606.29537 · 108 workflows, 500 steps, single tool call, max ✓xmodelOpus 4.8·max · Opus 4.7·max
OSWorld 2.0 single 150 steps
osworld-v2-board
osworld-v2.xlang.ai official JSON · 108 tasks, 150-step budget, Opus 4.7 max, Sonnet 4.6 medium/max ✓xmodelOpus 4.7·max
OSWorld 2.0 single 300 steps
osworld-v2-board
osworld-v2.xlang.ai official JSON · 108 tasks, 300-step budget, Opus 4.7 max, Sonnet 4.6 medium/max ✓xmodelOpus 4.7·max
OSWorld 2.0, offline set (OpenAI GPT-6 Sol and Luna post)
openai
OSWorld 2.0, offline set; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓xmodelOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
OSWorld-Pro
arxiv-2609.24890
arXiv 2609.24890 · OSWorld harness, 250 turns, 4 Claude models all at max, tokens × OpenRouter list incl. cache ✓xmodelOpus 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max
osworld2o5-bin
osworld-v2-board
—xmodelOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
osworld2o5a-bin
osworld-v2-board
—xmodelOpus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max
osworld2s150-bin
osworld-v2-board
—xmodelOpus 4.7·max
osworld2s300-bin
osworld-v2-board
—xmodelOpus 4.7·max
PA100 rubric verification, verifier AAAJ, Claude backbones
arxiv-2610.17724
all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier AAAJ; cost per full run; read 2026-10-15 ✓xmodelFable 5.1·high · Opus 5·high
PA100 rubric verification, verifier Archipelago, Claude backbones
arxiv-2610.17724
all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Archipelago; cost per full run; read 2026-10-15 ✓xmodelFable 5.1·high · Opus 5·high
PA100 rubric verification, verifier Autorubric, Claude backbones
arxiv-2610.17724
all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Autorubric; cost per full run; read 2026-10-15 ✓xmodelFable 5.1·high · Opus 5·high
PA100 rubric verification, verifier Gandalf, Claude backbones
arxiv-2610.17724
all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Gandalf; cost per full run; read 2026-10-15 ✓xmodelFable 5.1·high · Opus 5·high
Parallel Search Capability · with search
parallel.ai
Parallel Search Capability Leaderboard, Search Intelligence Score = mean of DSQA, HLE and WISER (100 questions each) with Parallel Search; cost per task incl. estimated search/extract usage; Claude 5.x at max effort per the page notes (Haiku 5.5 effort unstated); read 2026-10-15 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5·max
Parallel Search Capability · without search
parallel.ai
Same suites without the search tool (model alone); inference cost only; read 2026-10-15 ✓xmodelFable 5.1·max · Opus 5.5·max · Opus 5·max · Sonnet 5.5·max · Sonnet 5·max
ParseBench document-parsing leaderboard (run-llama/ParseBench)
run-llama/ParseBench
leaderboard.csv at commit afb36bd8 (2026-09-29); Overall score (Tables/Charts/ContentFaithfulness/SemanticFormatting/VisualGrounding average); cost is USD-cents per page from the repo's own pricing table (not independently verified against token counts, so flagged list-price-cost); n (documents) and run count not given in the csv; read 2026-10-01 ✓xmodelHaiku 4.5·nothink · Opus 4.8·nothink · Opus 5.5·high · Opus 5.5·low · Fable 5.1·high · Sonnet 5.5·high
pd-control
arxiv-2608.02670
—xmodelSonnet 5·high · Opus 4.8·high · Fable 5·high · Opus 5·high
pd-nisthigh
arxiv-2608.02670
—xmodelSonnet 5·high · Opus 4.8·high · Fable 5·high · Opus 5·high
pd-nonroot
arxiv-2608.02670
—xmodelSonnet 5·high · Opus 4.8·high · Fable 5·high · Opus 5·high
pebench-private
getcompound
—xmodelOpus 5.5·medium · Fable 5.1·high
PeterTh llm-eval-local: LLM autoparallelization of 220 programs (Claude Code, medium)
github-peterth-llm-eval-local
PeterTh/llm-eval-local analysis/tables/4d_all_models_score_vs_cost.csv; 220 programs per model-effort profile; slugs claude--cc-medium; score = mean overall score (0-10); cost = estimated USD per run from tokens x list price; read 2026-10-15 ✓xmodelHaiku 5.5·medium · Sonnet 5.5·medium · Sonnet 5·medium · Opus 5.5·medium · Fable 5·medium · Opus 5·medium
Phi-Bench (LLM infrastructure engineering, Full score), Claude Code, Opus 5 vs Sonnet 5
arxiv-2609.10226
arXiv 2609.10226 v2 (15 Oct 2026); Claude Code; Table 2 Full score; one H20 GPU/task; read 2026-10-18 ✓xmodelOpus 5·max · Sonnet 5·max
phi-Bench: file-system design and implementation, average pass rate
arxiv-2608.00280
Benchmarking LLMs on File System Design and Implementation (UIUC, arXiv 2608.00280, 4 Aug 2026) Figure 4 'Average' bars: pass rate % over six task types (BU/BI/PM/DE/PO/NF), 5 trials per task averaged; DIGITISED from the figure (src/img/phi_f4_zoom.png, pixel scale from the 60%/80% gridlines, +-1.5 points; script verify/phi_bars.py); Figure 10 labels give DeepSeek-V4-Flash 46 and Gemini-3.1-Pro 64 (different weighting); read 2026-10-20 ✓xmodelOpus 4.7·max
phyblas Tibetan-script reading web page (Vue), blind grading
qiita-phyblas
claude -p headless --effort medium, 3 models concurrently, 1 run each; blind grading by Opus 5.5 at high; 100 Wiktionary test words; cost = total_cost_usd (API pricing); read 2026-10-15 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Haiku 5.5·medium
PM-LLM-Benchmark v3, process-mining Q&A, judged by grok-4.7 (fit-alessandro-berti/pm-llm-benchmark3)
fit-alessandro-berti/pm-llm-benchmark3
leaderboard_grok-4.7.md at commit ab1371c (2026-09-30); 1-shot, 10-category score out of 60, LLM judge (grok-4.7, not a Claude judge); single run; no cost in this board; read 2026-10-01 ✓xmodelOpus 5·high · Fable 5·high
pokebench.tv: Pokemon Red speedrun to Brock, turns/time/cost
pokebench
fresh-save Pokemon Red, screenshot+RAM state per turn, budget 1000 turns, Anthropic API list-price cost; https://pokebench.tv/runs.json, src/pokebench_runs.json; read 2026-10-05 ✓xmodelOpus 5.5·high · Fable 5.1·high · Haiku 5.5·high · Fable 5.2·high · Sonnet 5.5·high
ponytail caveman arm
github-ponytail
promptfoo · 5 tasks ×30, caveman skill, cost only ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 4.8·nothink
ponytail skill arm
github-ponytail
promptfoo · 5 tasks ×30, ponytail skill, 3 Claude models ✓xmodelHaiku 4.5·nothink · Sonnet 4.6·nothink · Opus 4.8·nothink
PostTrainBench (GLM-5.2 card, evaluated by PostTrainBench)
zhipu
PostTrainBench, 1M context 128K output; board evaluation by posttrainbench.com; read 2026-10-19 ✓xmodelOpus 4.8·max
PostTrainBench v1.2 (weighted avg, 6 benchmarks x 4 base models)
posttrainbench
PostTrainBench leaderboard results v1.2 (posttrainbench.com/?version=v1.2, changelog 30 Sep / 6 Oct 2026): CLI agent post-trains Qwen3-1.7B, Qwen3-4B, SmolLM3-3B, Gemma-3-4B for 10 h on one H100; score = weighted mean of AIME25, Arena-Hard writing, GPQA Main, GSM8K, HealthBench, HumanEval (BFCL dropped), final evals averaged over up to 5 seeds, contamination judges majority of 3; mean over n runs (2-3); same-benchmark-as:posttrain (paper v1 snapshot); read 2026-10-20 ✓xmodelOpus 5.5·max · Opus 4.8·max · Opus 4.8·high · Opus 4.7·xhigh
prinzbench (private legal-research + search benchmark by a single author), leaderboard of 18 Jul 2026
github-prinz-ai-prinzbench
prinzbench (github.com/prinz-ai/prinzbench README 'Last updated on July 18, 2026'): 33 author-written questions (25 legal research + 8 needle search) asked 3 times, pass@1, graded by the author without blinding; score = full score x/99; access through chat apps or terminals at different dates (Jan-Jul 2026); '*' = early access before public release; the author cautions that Pro models use parallel test-time compute; read 2026-10-20 ✓xmodelFable 5·max · Opus 4.8·max
Professor Wiggum FlowER mechanism prediction, 10 hard cases, product given, one-shot baseline
github-professor-wiggum
scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓xmodelOpus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low
Professor Wiggum FlowER mechanism prediction, 10 hard cases, product given, Wiggum harness
github-professor-wiggum
scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓xmodelOpus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low
Professor Wiggum FlowER mechanism prediction, 10 hard cases, product hidden, one-shot baseline
github-professor-wiggum
scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓xmodelOpus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low
Professor Wiggum FlowER mechanism prediction, 10 hard cases, product hidden, Wiggum harness
github-professor-wiggum
scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓xmodelOpus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low
ProgramBench 166 (S55)
anthropic
Sonnet 5.5 card p118 · mini-swe-agent, 3 models, score-only ✓xmodelSonnet 5.5·max · Sonnet 5·max · Opus 5.5·max
ProgramDistill-300
arxiv-2609.18805
arXiv 2609.18805 · 300 web-app repair tasks, R2E-Gym harness, Opus 5 vs Sonnet 5 at high, tokens × LiteLLM list ✓xmodelOpus 5·high · Sonnet 5·high
ProofBench v1.0-era values (vals.ai, Epoch relay) · accuracy %
epoch.ai (vals.ai ProofBench v1.0-era)
ProofBench v1.0-era values relayed by Epoch AI (proofbench_external.csv, rows annotated 'v1.0-era value; model absent from the v1.1 board ... Not comparable to v1.1 rows'); vals.ai ProofBench v1.0 methodology (the held group valsproof is the v1.1 re-grade of 2026-08-14); cost per test as relayed; read 2026-10-20 ✓xmodelOpus 4.8·max · Opus 4.7·max
Protein library (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelSonnet 5·max · Fable 5.1·max · Opus 5.5·max
Protein sequence (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelSonnet 5.5·max · Fable 5.1·max · Opus 5.5·max
Protein trouble-shoot (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelFable 5.1·max · Opus 5.5·max
Protein understanding (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelFable 5.1·max · Opus 5.5·max
Qiita suwa_nobu cache-read
qiita-suwanobu
Opus 5 vs Opus 5.5 · same task, warm-cache rounds 2-3, CLI-metered cost ✓xmodelOpus 5·high · Opus 5.5·medium
Qiita suwa_nobu cache-write
qiita-suwanobu
Opus 5 vs Opus 5.5 · same task, cold-cache round, CLI-metered cost ✓xmodelOpus 5·high · Opus 5.5·medium
Qiita Takuya coding set (5 tasks ×3)
qiita-takuya
Opus 5.5 high/medium, Opus 5, Fable 5.1; hidden tests 100 % (saturated) ✓xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5·high · Fable 5.1·high
qiita-takuya-reason-hard
qiita-takuya
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5·high · Fable 5.1·high
qiita-takuya-reason-std
qiita-takuya
—xmodelOpus 5.5·high · Opus 5.5·medium · Opus 5·high · Fable 5.1·high
rails-orig-located
claude-maintain-rails-evals
—xmodelSonnet 5.5·low · Opus 5.5·medium
rails-v21-located
claude-maintain-rails-evals
—xmodelOpus 5.5·low
rampnet
github-rampnet
—xmodelSonnet 5·low · Sonnet 5·high · Opus 5·low · Opus 5·high
RampNet Fable
github-rampnet
same rig/prompt, effort=low, Anthropic 1P path · Fable 5 vs 5.1 ✓xmodelFable 5·low · Fable 5.1·low
reactbench
reactbench
—xmodelFable 5.1·max · Fable 5.1·xhigh · Fable 5·max · Fable 5·xhigh · Opus 5·high · Opus 5·max · Fable 5.1·high · Fable 5·high · Opus 5·xhigh · Opus 5·medium · Fable 5·low · Opus 4.8·max · Opus 4.8·xhigh · Sonnet 5·xhigh · Sonnet 5·max · Fable 5.1·medium · Sonnet 5·high · Fable 5.1·low · Opus 4.8·medium · Opus 5·low · Opus 4.8·high · Sonnet 5·medium · Opus 4.8·low · Sonnet 5·low · Opus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low
reactbench-fix
reactbench
—xmodelFable 5.1·max · Opus 5·high · Fable 5.1·xhigh · Fable 5·xhigh · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5·low · Fable 5.1·medium · Fable 5·high · Fable 5·max · Opus 4.8·medium · Sonnet 5·max · Opus 4.8·max · Opus 4.8·xhigh · Opus 5·low · Opus 4.8·high · Sonnet 5·high · Sonnet 5·medium · Opus 4.8·low · Sonnet 5·low · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·max · Opus 5.5·low
reactbench-write
reactbench
—xmodelFable 5·max · Fable 5.1·max · Fable 5.1·xhigh · Fable 5.1·high · Fable 5·xhigh · Opus 5·max · Opus 5·high · Fable 5·high · Opus 5·xhigh · Opus 5·medium · Fable 5·low · Opus 4.8·max · Opus 4.8·xhigh · Sonnet 5·high · Sonnet 5·max · Fable 5.1·medium · Sonnet 5·xhigh · Fable 5.1·low · Opus 4.8·high · Opus 4.8·medium · Opus 5·low · Opus 4.8·low · Sonnet 5·medium · Sonnet 5·low · Opus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low
Reader-highlight sentence ranking (215 dev pages), zero-shot
arxiv-2610.17768
average-precision difference vs the cached gemini-3.1-pro-preview reference, first sample, one scoring run; low reasoning effort; read 2026-10-15 ✓xmodelOpus 5.5·low · Fable 5.1·low
RealityBench acceptability (1,378 frames)
arxiv-2610.14397
zero-shot VLM acceptability F1 against the rule scorer; cost = whole set at developer list price, thinking at output rate; read 2026-10-15 ✓xmodelFable 5.1·high · Opus 5·high
Reddit 1x0poki (smith2008): Three Little Pigs SVG film, blind cross-judged
reddit-smith2008
20-s film of 600 SVG frames, each model in Claude Code, one run; each model scored all three films blind (frame sheets); score /10; read 2026-10-15 ✓xmodelSonnet 5.5·high · Opus 5.5·high · Haiku 5.5·high
Reddit 1x132yy (Escobar747): cross-language agentic task in a real Rust repo
reddit-Escobar747
one agentic task (move logic across a language boundary following repo conventions); cost per task as self-reported (Sonnet: at list prices); qualitative score only; read 2026-10-15 ✓xmodelHaiku 5.5·medium · Haiku 5.5·high
Reddit 1x1jbry (AlexXx52): one read-only Unity investigation, subscription allowance
reddit-AlexXx52
same read-only investigation of a Unity 6 project on each consumer subscription (Claude Pro for Claude); 'cost' = % of the subscription's own usage allowance (NOT USD, not comparable across providers); score = author's subjective quality /10; 1 run; read 2026-10-15 ✓xmodelSonnet 5.5·medium · Opus 5.5·medium · Opus 4.8·medium · Sonnet 5.5·high · Opus 5.5·high · Opus 4.8·high · Sonnet 4.6·high
Reddit 1x1nsaz (emarkosov): 5 web tasks (3 research + 2 fact-check) cost
reddit-emarkosov
5 web tasks as subagents, high effort; cost for all 5 tasks as self-reported; read 2026-10-15 ✓xmodelHaiku 5.5·high · Haiku 5.5·xhigh
Reddit 1x1nsaz (emarkosov): code search task, Haiku/Sonnet/Opus 5.5 as subagents
reddit-emarkosov
re-run of a real task (find every file for a feature; 48 true files) on each model as subagent, high effort; score = files found of 48; cost as self-reported; read 2026-10-15 ✓xmodelOpus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high · Haiku 5.5·xhigh
reddit-1xsuixt-pelican-go-all-out
reddit
—xmodelFable 5.2·low · Opus 5.5·medium
reddit-1xtq03o-md-rewrite-quality-norm
reddit
—xmodelFable 5.2·low · Opus 5.5·medium · Fable 5.2·medium · Opus 5.5·high · Fable 5.2·high · Opus 5.5·xhigh · Fable 5.2·xhigh · Opus 5.5·max
ReLE 专业能力 (professional ability)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/专业能力.md (专业能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 医疗与心理健康 (medical/mental health)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/医疗与心理健康.md (医疗与心理健康); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 总分 (total)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/total.md (alldata); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 推理与数学计算 (reasoning/math)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/推理与数学计算.md (推理与数学计算); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 教育 (education)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/教育.md (教育); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 法律与行政公务 (law/admin)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/法律与行政公务.md (法律与行政公务); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 语言与指令遵从 (language/IF)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/语言与指令遵从.md (语言与指令遵从); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 通用能力 (general ability)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/通用能力.md (通用能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
ReLE 金融 (finance)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/金融.md (金融); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelOpus 5·high · Sonnet 5.5·high · Opus 4.8·nothink
renchris review s55 + o55 rejudge
github-renchris-infra
claude -p no-tools, CC 2.1.284 · same 9-brief corpus as renchris-o55 (7 scored / 36 anchored defects), Sonnet 5.5 low→max (new) judged in the same pass as a re-judging of the held Opus 5.5 medium/high/xhigh outputs; 3 blind judges, strict majority; CLI list cost for the Sonnet 5.5 cells only ✓; read 2026-10-01 ✓xmodelSonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5·max · Fable 5.1·high
renchris synthesis-worker probe (Sonnet 5.5 vs Opus 5.5 xhigh)
github-renchris-infra
6 paired synthesis briefs, 130 key items, 1 cell per arm/brief; judges 2x Opus 5.5 + 1x Opus 5 (claude-judge); mean score /10 over all 18 judge rows; cost/tokens summed over the 6 cells; read 2026-10-01 ✓xmodelSonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Opus 5.5·xhigh
renkon40 · 50-mail triage
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, 50-mail triage (49 scored); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓xmodelOpus 5.5·high · Fable 5.1·high
renkon40 · sycophancy test, 20 questions
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, sycophancy test, 20 questions; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓xmodelOpus 5.5·high · Fable 5.1·high · Opus 5·high · Fable 5·high · Sonnet 5.5·high
Retort brazil-soccer-mcp (clojure): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (cpp): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (csharp): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (elixir): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (erlang): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (go): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (java): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (objc): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (python): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (rust): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (swift): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort brazil-soccer-mcp (typescript): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (c): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (clojure): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (cpp): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (elixir): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (erlang): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (java): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (objc): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (rust): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (swift): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
Retort rest-api-crud (typescript): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86)
github-retort
retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓xmodelOpus 5.5·low · Haiku 5.5·low
roblox-leetcode-easy
hf-roblox-llm-leaderboard
—xmodelFable 5·medium · Fable 5.1·medium · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium
roblox-leetcode-hard
hf-roblox-llm-leaderboard
—xmodelFable 5·medium · Fable 5.1·medium · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium
roblox-leetcode-medium
hf-roblox-llm-leaderboard
—xmodelFable 5·medium · Fable 5.1·medium · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium
roblox-qa-freeform
hf-roblox-llm-leaderboard
—xmodelFable 5·medium · Fable 5.1·medium · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium
roblox-qa-multiple-choice
hf-roblox-llm-leaderboard
—xmodelFable 5·medium · Fable 5.1·medium · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium
robocolis-agent2act
arxiv-2610.27865
—xmodelOpus 5.5·high
Roboflow Playground Vision Evals, OCR (low tier)
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: OCR; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-ocr) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓xmodelOpus 5.5·low · Sonnet 5.5·low · Fable 5.1·low · Fable 5·low · Opus 5·low · Haiku 5.5·low · Opus 4.8·low · Sonnet 5·low
Roboflow Playground Vision Evals, overall (mean of tasks) with cost per sample
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; Overall = page's mean over the tasks scored (4/5 where OCR is missing); cost = 'Avg cost / sample' in USD from price x tokens at the pricing date 2026-10-19 ('~' = estimated from self-host/OpenRouter pricing); read 2026-10-20 ✓xmodelOpus 5.5·low · Sonnet 5.5·low · Fable 5.1·low · Fable 5·low · Opus 5·low · Haiku 5.5·low · Opus 4.8·low · Sonnet 5·low
RoboQuest (10 tasks, 50 episodes per cell), Inspect Robots harness
arxiv-2610.10388
200-decision budget per episode, success rate over 50 episodes x 10 tasks; cost per episode at list prices; read 2026-10-15 ✓xmodelOpus 5.5·medium · Fable 5.1·medium
RoTA-Bench (interactive 3D projection alignment, 100 tasks)
arxiv-2610.17665
RoTA-Bench (arXiv 2610.17665, 13 Oct 2026): 100 tasks, interactive 3D projection alignment, 48 decisions max, same prompt/harness for every model; score = success % (IoU >= 0.9; Final IoU and steps are other columns); cost = Table 20 total USD over the 100 tasks estimated by the authors from recorded tokens and published list prices; Temperature 0, reasoning effort medium where supported; read 2026-10-20 ✓xmodelSonnet 5·medium · Opus 5·medium
rotabench-final-iou
arxiv-2610.17665
—xmodelSonnet 5·medium · Opus 5·medium
rt-v1.1-text-sc
heydonna
—xmodelSonnet 5.5·high
rtk-base
arxiv-2607.12161
—xmodelSonnet 5·low · Sonnet 5·high · Opus 4.8·low · Opus 4.8·high
rtx4090-local-vs-cloud: 8 manufacturing-domain tasks, programmatic grading, cloud Claude rows via Claude Code
github-rtx4090-local-vs-cloud
Howard159357/rtx4090-local-vs-cloud-coding-benchmark README (15 Oct 2026); 3 runs per setup, Claude Code CLI with built-in shell/web/plugins disabled; subscription (no API cost); score = mean of 3 runs as printed; read 2026-10-15 ✓xmodelSonnet 5.5·medium · Haiku 5.5·medium · Haiku 5.5·xhigh
rubench-claude-code
arxiv-2607.06411
—xmodelOpus 4.8·xhigh · Sonnet 5·xhigh
RuneBench
github-runebench
Harbor+Modal · 16 skills, 30-min budget, XP metric; mispriced Opus 5 cells omitted ✓xmodelFable 5·xhigh · Fable 5.1·xhigh · Fable 5·high · Fable 5.1·high · Opus 4.7·xhigh · Opus 4.8·high · Opus 4.8·max · Sonnet 5·xhigh · Sonnet 5.5·medium · Sonnet 5.5·xhigh · Haiku 5.5·high
s7t-bsky-ironcurtain-fps-gun
bluesky
—xmodelFable 5.2·xhigh · Opus 5.5·xhigh
s7t-claudebattle-auric-final
github-eval-forge
—xmodelFable 5.2·high · Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high
s7t-claudebattle-auric-quality
github-eval-forge
—xmodelFable 5.2·high · Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high
s7t-claudebattle-auric-truthness
github-eval-forge
—xmodelFable 5.2·high · Opus 5.5·high · Haiku 5.5·high
s7t-nbl-mnist-step1
notesbylex
—xmodelFable 5.2·high · Fable 5.2·low · Fable 5.1·high · Opus 5.5·high
s7t-nbl-mnist-step15
notesbylex
—xmodelFable 5.2·high · Fable 5.2·low · Fable 5.1·high · Opus 5.5·high
s7t-techkwon-motion
github-techkwon
—xmodelOpus 5.5·high
s7t-techkwon-physics
github-techkwon
—xmodelOpus 5.5·high
s7t-techkwon-svg
github-techkwon
—xmodelOpus 5.5·high
s7t-techkwon-three
github-techkwon
—xmodelOpus 5.5·high
s7t-wsp-exp5-seaquest
huggingface-shawnghu
—xmodelFable 5.2·high · Fable 5.2·xhigh
sakana-rhi47
arxiv-2607.15524
—xmodelOpus 4.7·high · Opus 4.7·max · Opus 4.7·xhigh
sakana-rhi48
arxiv-2607.15524
—xmodelOpus 4.8·high · Opus 4.8·max · Opus 4.8·xhigh
Scale EnigmaEval (Wayback 2026-10-16, current 5-model board)
scale-seal
https://labs.scale.com/leaderboard/enigma_eval; Wayback capture 20261016182047 (saved gpt-5.1-chat-latest/src/scale/dec_enigma_eval.html); score = board score % with 95% CI; puzzle hunts; current board (models listed with a -high / -xhigh suffix); read 2026-10-20 ✓xmodelFable 5·high · Opus 4.8·xhigh
Scale Humanity's Last Exam text-only (Wayback 2026-10-07)
scale-seal
https://labs.scale.com/leaderboard/humanitys_last_exam_text_only; Wayback capture 20261007135520 (saved gpt-5.1-chat-latest/src/scale/dec_humanitys_last_exam_text_only.html); score = board score % with 95% CI; HLE text-only, board page of 2026-10-07 (HLE-Rolling note of 17 Sep 2026); read 2026-10-20 ✓xmodelFable 5.1·xhigh
SCBench
scbench.ai
SCBench, agentic coding through successive checkpoints in Claude Code · strict solve rate in %, cost per checkpoint (USD); effort from the model label (High); the Claude Code version differs per row; read 2026-09-30 ✓xmodelOpus 5.5·medium · Opus 5.5·xhigh · Fable 5.1·xhigh
scbench-iso
scbench.ai
—xmodelOpus 5.5·medium · Opus 5.5·xhigh · Fable 5.1·xhigh
SCDBench smart-contract decompilation, with one zero-shot compilation-repair pass (dagger rows)
arxiv-2605.29059
SCDBench (arXiv 2605.29059, 27 May 2026): 600 real Solidity contracts (200 easy/medium/hard), with one zero-shot compilation-repair pass (dagger rows); score = semantic consistency (% of 14,553 public functions passing replayable semantic checkpoints; Table 4; the earlier stages format/compile/ABI are other cumulative columns); cost = Table 9 average API cost per contract at OpenRouter prices at the time; read 2026-10-20 ✓xmodelOpus 4.7·high
SCDBench smart-contract decompilation, zero-shot decompilation
arxiv-2605.29059
SCDBench (arXiv 2605.29059, 27 May 2026): 600 real Solidity contracts (200 easy/medium/hard), zero-shot decompilation; score = semantic consistency (% of 14,553 public functions passing replayable semantic checkpoints; Table 4; the earlier stages format/compile/ABI are other cumulative columns); cost = Table 9 average API cost per contract at OpenRouter prices at the time; read 2026-10-20 ✓xmodelOpus 4.7·high
scf52-arc3p
anthropic
—xmodelFable 5.2·high · Fable 5.2·max
scf52-arc3s
anthropic
—xmodelFable 5.2·high · Fable 5.2·max
scf52-autom
anthropic
—xmodelFable 5.2·low · Fable 5.2·max · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Opus 5.5·max
scf52-bcadn
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh
scf52-bcadt
anthropic
—xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5.2·max
scf52-briefe
anthropic
—xmodelFable 5.2·max · Fable 5.2·xhigh
scf52-chartn
anthropic
—xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.2·max · Opus 5.5·max · Fable 5.1·max
scf52-charto
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Sonnet 5.5·max
scf52-chartr
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Sonnet 5.5·max
scf52-chartt
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh
scf52-cve
anthropic
—xmodelOpus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max
scf52-draco
anthropic
—xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max
scf52-fc20
anthropic
—xmodelFable 5.2·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh
scf52-fswe2
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max
scf52-gdpval
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max · Fable 5.2·xhigh
scf52-gmmlu
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max
scf52-hb
anthropic
—xmodelFable 5.2·max · Fable 5.2·low · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Haiku 5.5·max
scf52-hba
anthropic
—xmodelFable 5.2·max · Fable 5.2·low · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Haiku 5.5·max
scf52-hbp
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Haiku 5.5·max
scf52-hbpa
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Haiku 5.5·max
scf52-hlen
anthropic
—xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max
scf52-hlet
anthropic
—xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max
scf52-milu
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max
scf52-offqa
anthropic
—xmodelFable 5.2·max · Fable 5.1·max · Opus 5.5·max
scf52-offqap
anthropic
—xmodelFable 5.2·max · Fable 5.1·max · Opus 5.5·max
scf52-oswp
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh
scf52-osws
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max · Fable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh
scf52-phys
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5.2·low · Fable 5.2·high · Fable 5.2·xhigh
scf52-rli
anthropic
—xmodelFable 5.2·high · Opus 5.5·high · Fable 5.1·high
scf52-robodojo
anthropic
—xmodelFable 5.1·max · Opus 5.5·max · Fable 5.2·max
scf52-stemprof
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max
scf52-swepro-sub
anthropic
—xmodelOpus 5.5·high · Opus 5.5·low
scf52-tb30
anthropic
—xmodelFable 5.1·max · Opus 5.5·high
scf52-tb4
anthropic
—xmodelFable 5.2·max · Opus 5.5·xhigh · Fable 5.1·max
scf52-tbsci
anthropic
—xmodelFable 5.2·max · Opus 5.5·max · Fable 5.1·max
scf52-wandr
anthropic
—xmodelFable 5.2·low · Fable 5.2·medium · Fable 5.2·high · Fable 5.2·xhigh · Fable 5.2·max
schletools
anthropic
—xmodelSonnet 5·max · Opus 4.8·max · Sonnet 4.6·low · Sonnet 4.6·high
Search agent with gold passages kept, answer EM on HOTPOTQA (vanilla retrieval tool)
arxiv-2610.05348
300 HotpotQA questions, BM25 over wiki-18, vanilla tool; Table 1/4 of arXiv 2610.05348; read 2026-10-15 ✓xmodelSonnet 5.5·high
Search agent with gold passages kept, answer EM on NQ (vanilla retrieval tool)
arxiv-2610.05348
257 NQ questions, BM25 over wiki-18, vanilla tool; Table 1/4 of arXiv 2610.05348; read 2026-10-15 ✓xmodelSonnet 5.5·high
Senko Flysim
senko
Senko Rašić · one-shot flight-sim game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓xmodelOpus 5.5·xhigh · Opus 5·xhigh · Haiku 5.5·xhigh
Senko RTS
senko
Senko Rašić · one-shot RTS game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓xmodelOpus 5.5·xhigh · Opus 5·xhigh · Haiku 5.5·xhigh
Senko Voxel
senko
Senko Rašić · one-shot Minecraft-like game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓xmodelOpus 5.5·xhigh · Opus 5·xhigh · Haiku 5.5·xhigh
SenseBench (WSD) · prompt p001
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelOpus 5·max · Opus 4.8·xhigh · Opus 4.8·high · Opus 4.8·nothink · Opus 4.8·low · Opus 4.8·medium · Opus 4.7·medium · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5·high · Sonnet 4.6·low · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh · Opus 5.5·low · Opus 5.5·xhigh · Sonnet 5.5·low · Sonnet 5.5·xhigh
SenseBench (WSD) · prompt p002
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p002 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelOpus 4.8·high · Sonnet 4.6·low · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh
SenseBench (WSD) · prompt p003
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p003 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelOpus 4.7·medium · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh
SenseBench (WSD) · prompt p004
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p004 (new prompt variant) · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-10-15 ✓xmodelHaiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh · Opus 4.8·xhigh
ServeLearnBench Avg. Hidden (Blind)
arxiv-2610.07792
ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Blind; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓xmodelOpus 5·high
ServeLearnBench Avg. Hidden (CH)
arxiv-2610.07792
ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness CH; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓xmodelOpus 5·high
ServeLearnBench Avg. Hidden (Mem0)
arxiv-2610.07792
ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Mem0; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓xmodelOpus 5·high
ServeLearnBench Avg. Hidden (Oracle)
arxiv-2610.07792
ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Oracle; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓xmodelOpus 5·high
ServeLearnBench Avg. Hidden (RAG)
arxiv-2610.07792
ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness RAG; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓xmodelOpus 5·high
ServeLearnBench Avg. Hidden (SkillOpt)
arxiv-2610.07792
ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness SkillOpt; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓xmodelOpus 5·high
SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies): MCC on primary studies
sesr-eval-2610.10633
SESR-Eval screening of SE systematic reviews (SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies)): LLM title/abstract screening via OpenRouter + Pydantic AI, MCC pooled over primary studies (Accuracy/Precision/Recall/F1/lost evidence and the secondary-study table are other summaries of the same runs: rule 8); cost = USD per secondary study (5% sample) from the secondary-study table; Claude 3.7 Sonnet is an old model whose cost is a previously recorded estimate; arXiv 2610.10633 (Oct 2026); read 2026-10-20 ✓xmodelOpus 4.7·low
Single-cell (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelFable 5.1·max · Opus 5.5·max
singularitygate-v2-1-all-80
singularitygate
—xmodelFable 5.2·xhigh · Opus 5.5·xhigh · Fable 5.1·xhigh · Sonnet 5.5·xhigh · Haiku 5.5·xhigh
singularitygate-v2-1-hard-12
singularitygate
—xmodelFable 5.2·xhigh · Opus 5.5·xhigh · Fable 5.1·xhigh · Sonnet 5.5·xhigh · Haiku 5.5·xhigh
Spatial transcriptomics (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelFable 5.1·max · Opus 5.5·max
spatialscout-micro-all
arxiv-2610.26659
—xmodelOpus 5.5·xhigh · Sonnet 5.5·xhigh
SpireBench: Slay the Spire Ascension 20 Ironclad, mean final floor reached
arxiv-2610.12596
10 runs per model on the same held-out seeds, themed reskin with note-taking; Table 1 and Table 26 of arXiv 2610.12596; read 2026-10-15 ✓xmodelFable 5.1·low · Fable 5.1·max · Opus 5·low · Opus 5·max · Sonnet 5·low · Sonnet 5·max
stet-board24-oct
stet.sh
—xmodelOpus 5·medium · Haiku 5.5·max · Haiku 5.5·low · Opus 4.8·medium
stet-pr24
stet.sh
—xmodelSonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max
Supabase evals (with skills)
supabase.com
Supabase evals in Claude Code, with skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓xmodelOpus 5.5·high · Sonnet 5·high
Supabase evals (without skills)
supabase.com
Supabase evals in Claude Code, without skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓xmodelOpus 5.5·high · Sonnet 5·high
SuperCLUE 2026年5月 代码生成
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelOpus 4.8·high · Opus 4.7·high
SuperCLUE 2026年5月 幻觉控制
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelOpus 4.8·high · Opus 4.7·high
SuperCLUE 2026年5月 数学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelOpus 4.8·high · Opus 4.7·high
SuperCLUE 2026年5月 智能体(任务规划)
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelOpus 4.8·high · Opus 4.7·high
SuperCLUE 2026年5月 科学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelOpus 4.8·high · Opus 4.7·high
SuperCLUE 2026年5月 精确指令遵循
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelOpus 4.8·high · Opus 4.7·high
Surface Evolver bench
yhenon-surface-evolver
yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓xmodelFable 5·high · Opus 4.8·high · Sonnet 5·medium
surfevolver-pass
yhenon-surface-evolver
—xmodelFable 5·high · Opus 4.8·high · Sonnet 5·medium
Surge AI · Antidote: Everyday Edition
surgehq.ai
Surge AI Antidote: Everyday Edition leaderboard (surgehq.ai/benchmarks/antidote); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelFable 5·high · Fable 5.1·high · Opus 5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 4.6·high
Surge AI · Chartography
surgehq.ai
Surge AI Chartography leaderboard (surgehq.ai/benchmarks/chartography); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max
Surge AI · ComplexConstraints
surgehq.ai
Surge AI ComplexConstraints leaderboard (surgehq.ai/benchmarks/complex-constraints); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max
Surge AI · DAYJOB: Finance
surgehq.ai
Surge AI DAYJOB: Finance leaderboard (surgehq.ai/benchmarks/dayjob-finance); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max
Surge AI · DAYJOB: Healthcare
surgehq.ai
Surge AI DAYJOB: Healthcare leaderboard (surgehq.ai/benchmarks/dayjob-healthcare); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max
Surge AI · EnterpriseBench: CoreCraft
surgehq.ai
Surge AI EnterpriseBench: CoreCraft leaderboard (surgehq.ai/benchmarks/enterprisebench-corecraft); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelFable 5.1·max · Opus 5.5·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Sonnet 4.6·high
Surge AI · GDP.pdf
surgehq.ai
Surge AI GDP.pdf leaderboard (surgehq.ai/benchmarks/gdp-pdf); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelSonnet 5.5·max · Opus 5.5·max · Fable 5·max · Fable 5.1·max · Opus 4.8·max · Opus 5·max · Opus 4.7·max
Surge AI · GDP.xlsx
surgehq.ai
Surge AI GDP.xlsx leaderboard (surgehq.ai/benchmarks/gdp-xlsx); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Sonnet 5·max
Surge AI · HANDBOOK.md Agents
surgehq.ai
Surge AI HANDBOOK.md Agents leaderboard (surgehq.ai/benchmarks/handbook); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelFable 5·max · Opus 5·max · Opus 4.8·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max
Surge AI · Hemingway-bench
surgehq.ai
Surge AI Hemingway-bench leaderboard (surgehq.ai/benchmarks/hemingway-bench); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelFable 5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Opus 4.7·high · Sonnet 4.6·high
Surge AI · Riemann-bench
surgehq.ai
Surge AI Riemann-bench leaderboard (surgehq.ai/benchmarks/riemann-bench); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max
Surge AI · sudo L7
surgehq.ai
Surge AI sudo L7 leaderboard (surgehq.ai/benchmarks/sudo-l7); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Fable 5.1·max
surge-twi-oct9
surge-twi
—xmodelFable 5·max
SWE Atlas Fig. 14: cost per task vs Pass@1 on QnA + Test Writing
scale-swe-atlas
SWE Atlas Appendix G Figure 14 (arXiv 2605.08366): per-trial token usage from the scaffold logs x each provider's published per-million-token API rates; 642 trials per model (372 Q&A + 270 Test Writing); native vendor scaffold; both axes DIGITISED from vector markers (axes calibrated on tick marks, residual 2e-5 pt); price date not given; read 2026-10-19 ✓xmodelOpus 4.7·xhigh
SWE Bench Pro (xAI Grok 4.5 post)
xai
SWE Bench Pro resolve rate %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; output tokens per task for Grok 4.5 and Opus 4.8 from the caption; read 2026-10-19 ✓xmodelOpus 4.8·max · Opus 4.7·max
SWE Marathon (xAI Grok 4.5 post)
xai
SWE Marathon resolution rate pass@1 %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; read 2026-10-19 ✓xmodelOpus 4.8·max · Opus 4.7·max
SWE-bench Multilingual (S55)
anthropic
Sonnet 5.5 card table 8.1.A · 3 models at max, score-only ✓xmodelSonnet 5.5·max · Sonnet 5·max · Opus 5.5·max
SWE-bench Multimodal (S55)
anthropic
Sonnet 5.5 card table 8.1.A · 3 models at max, score-only ✓xmodelSonnet 5.5·max · Sonnet 5·max · Opus 5.5·max
SWE-bench Pro (cards)
anthropic
Sonnet 5 card and Sonnet 5.5 card table 8.1.A · 5 Claude models at max, score only ✓xmodelSonnet 5.5·max · Opus 5.5·max · Sonnet 5·max
SWE-Bench Pro V2 (Scale)
scale-seal
Scale Labs board via Wayback 23 Sep 2026 · full set, Claude Code, one effort per model, score only, saturated ✓xmodelOpus 5·xhigh · Fable 5.1·high · Sonnet 5·xhigh
SWE-Bench Pro V2 HARD (Scale)
scale-seal
Scale Labs board via Wayback 23 Sep 2026 · HARD subset, Claude Code, score only ✓xmodelOpus 5·xhigh · Fable 5.1·high · Sonnet 5·xhigh
SWE-Marathon (GLM-5.2 card, evaluated by Abundant AI)
zhipu
SWE-Marathon, 1M context 128K output; evaluation by Abundant AI; read 2026-10-19 ✓xmodelOpus 4.8·max
swe-rebench 07/26
swe-rebench
ReAct minimal · Opus5/Fable5/Sonnet5 all at high ✓xmodelFable 5·high · Opus 5·high · Sonnet 5·high
synthbench-althing-globalopinionqa
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·nothink · Sonnet 4.6·nothink · Sonnet 5.5·high
synthbench-althing-gss
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·nothink · Sonnet 4.6·nothink · Sonnet 5.5·high
synthbench-althing-opinionsqa
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·nothink · Sonnet 4.6·nothink · Sonnet 5.5·high
synthbench-althing-subpop
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·nothink · Sonnet 4.6·nothink · Sonnet 5.5·high
synthbench-plain-globalopinionqa
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·low · Sonnet 4.6·nothink · Sonnet 5.5·low
synthbench-plain-gss
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·low · Sonnet 4.6·nothink · Sonnet 5.5·low
synthbench-plain-opinionsqa
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·low · Sonnet 4.6·nothink · Sonnet 5.5·low
synthbench-plain-subpop
github-synthbench
—xmodelHaiku 4.5·nothink · Haiku 5.5·low · Sonnet 4.6·nothink · Sonnet 5.5·low
Synthorai (ja mirror): 4-question tool-reading loop, Sonnet 5.5 (sweep) and Opus 5.5 default
synthorai
native Messages API via gateway, 4 tasks x3 reps per row (n=12), cost computed from list price; qiita.com/Synthorai/items/98e776a66f483f3ceeaf (mirror of synthorai.io/ja/blog/claude-sonnet-5-5-cost/); read 2026-10-05 ✓xmodelSonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·max · Sonnet 5·high
Takuya__ (qiita, item 2a513efa) 12 sqlite-utils bug-fix/feature tasks (incl 4 real upstream issues), x2 runs, full test suite
qiita-takuya55
Claude Code 2.1.293, Read/Edit/Write/Grep/Glob/Bash, pass = all hidden+existing tests (~1000) green; read 2026-10-08 ✓xmodelHaiku 5.5·medium · Haiku 5.5·high · Sonnet 5.5·medium
Takuya__ (qiita, item 2a513efa) Banking77 intent classification + "narrow candidates first" prompt addition
qiita-takuya55
same task as qiita-takuya55-banking77 but with an added system-prompt instruction to narrow candidates first: different configuration; read 2026-10-08 ✓xmodelHaiku 5.5·medium · Sonnet 5.5·medium
Takuya__ (qiita, item 2a513efa) Banking77 intent classification, 154 msgs/77 labels, no tools, single API call
qiita-takuya55
no-tools system-prompt label list, single API call per message, cost computed from measured tokens x price table per 1,000 msgs; read 2026-10-08 ✓xmodelHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max · Sonnet 5.5·medium
Takuya__ (qiita, item 2a513efa) code-reading Q&A, sqlite-utils 3.38, 9 questions x2 runs
qiita-takuya55
Claude Code 2.1.293 claude -p stream-json, Read/Grep/Glob/Bash only, cost = measured tokens x official price table; read 2026-10-08 ✓xmodelHaiku 5.5·low · Haiku 5.5·medium · Sonnet 5.5·medium
Takuya__ (qiita, item 2a513efa) Sonnet-lead orchestration, 9 Q x3 runs, pure single-model baselines only
qiita-takuya55
Claude Code 2.1.293, Explore subagent delegation; only the two PURE single-model runs are rows (mixed Sonnet-lead+Haiku-Explore run excluded: not a model x effort couple); read 2026-10-08 ✓xmodelSonnet 5.5·medium · Haiku 5.5·medium
TashonBraganca/messy-docs-bench: field-extraction accuracy, Opus 5.5 vs Sonnet 5
github-messydocsbench
137 messy documents (receipts/invoices/scans/W-2/statements/contracts), field-level extraction accuracy; github.com/TashonBraganca/messy-docs-bench, src/messydocs_summary.json; read 2026-10-05 ✓xmodelOpus 5.5·medium · Sonnet 5·high
TasteVal compute multiplier vs expert baseline (Researcher models, 8 ML-research tasks)
arxiv-2610.06824
8 uncontaminated AI R&D tasks, 6 seeds/model/task, 40 H100-hours; Researcher model + fixed Coder (Opus 4.8); Table 2 of arXiv 2610.06824; score only (cost in round 2); read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 4.8·max · Opus 4.7·max
TasteVal performance multiplier vs expert baseline
arxiv-2610.06824
8 uncontaminated AI R&D tasks, 6 seeds/model/task, 40 H100-hours; Researcher model + fixed Coder (Opus 4.8); Table 2 of arXiv 2610.06824; score only (cost in round 2); read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 4.8·max · Opus 4.7·max
TaxCalcBench TY25, no web search (column-tax/tax-calc-bench)
column-tax-tax-calc-bench
50 TY25 federal+state tax-return cases, 1 run each; strict-correct-return % is the score (lenient% also in the evidence but not used here per instructions); cost = mean litellm-estimate USD/return (n priced noted when <50); source: taxcalc_ty25_claude_by_effort.txt, re-verified against raw evaluation_result_*.md for opus-5.5/ultrathink (40.00%/50.00%/$1.135, exact match); read 2026-10-01 ✓xmodelOpus 5.5·max · Fable 5.1·max · Opus 5.5·high · Opus 5.5·xhigh · Fable 5·xhigh · Fable 5.1·xhigh · Fable 5.1·high · Fable 5·high · Opus 5.5·medium · Fable 5.1·medium · Opus 5·xhigh · Opus 4.8·xhigh · Sonnet 5.5·xhigh · Opus 5·max · Opus 5·high · Fable 5·medium · Opus 5·medium · Fable 5.1·low · Sonnet 5.5·high · Opus 4.8·medium · Opus 5.5·low · Opus 4.8·high · Fable 5·low · Opus 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Sonnet 5·xhigh · Haiku 5.5·high · Haiku 5.5·medium · Opus 4.8·low · Haiku 5.5·low
TaxCalcBench TY25, with web search (column-tax/tax-calc-bench)
column-tax-tax-calc-bench
Same as taxcalc-ty25-notool but with the web-search tool enabled; read 2026-10-01 ✓xmodelOpus 5.5·max · Fable 5.1·xhigh · Opus 5.5·xhigh · Fable 5.1·max · Opus 5·max · Sonnet 5.5·max · Fable 5.1·high · Fable 5·max · Opus 5.5·high · Sonnet 5.5·xhigh · Fable 5·xhigh · Opus 5·xhigh · Opus 4.8·xhigh · Opus 5.5·medium · Fable 5·high · Opus 5·high · Fable 5.1·medium · Sonnet 5.5·high · Opus 4.8·max · Haiku 5.5·xhigh · Fable 5·medium · Fable 5·low · Fable 5.1·low · Opus 5·medium · Opus 4.8·high · Opus 5.5·low · Opus 4.8·medium · Haiku 5.5·high · Opus 5·low · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·medium · Haiku 5.5·low
TB-Science 0.1 (S55)
anthropic
Sonnet 5.5 card p114 · Claude Code --bare, 5 models at max, score-only ✓xmodelSonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max
tb21cc
tbench
—xmodelOpus 4.8·high · Sonnet 5·high
TCSAlgBench 10-round discussion acceptance % (calibration runs)
arxiv-2609.35606
TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); separate runs of Table 9 (acceptance %, RMS-CE reported separately); read 2026-10-19 ✓xmodelOpus 4.8·xhigh · Opus 4.8·high · Opus 4.8·max
TCSAlgBench 10-round discussion, 5-run coverage
arxiv-2609.35606
TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); 10-round discussion, coverage over 5 runs; read 2026-10-19 ✓xmodelOpus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·high · Fable 5·xhigh
TCSAlgBench direct inference, 10-run coverage
arxiv-2609.35606
TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); direct inference, coverage = challenges accepted in at least one of 10 runs; read 2026-10-19 ✓xmodelOpus 4.8·max · Opus 4.8·xhigh · Opus 4.8·high · Fable 5·xhigh · Fable 5·high
techkwon-token-diet-lesson
github-techkwon
—xmodelFable 5.2·high · Fable 5.2·low · Opus 5.5·high
techkwon-token-diet-motion
github-techkwon
—xmodelFable 5.2·high · Fable 5.2·low · Opus 5.5·high
Terminal Bench 2.1 (xAI Grok 4.5 post)
xai
Terminal Bench 2.1 score %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; read 2026-10-19 ✓xmodelOpus 4.8·max · Opus 4.7·max
Terminal-Bench 3.0 (GLM-5.3 card)
zhipu
Terminal-Bench-3, Claude Code 2.1.207, avg@3, 600 turns, 10h timeout, Tool Search disabled, official verifier; GLM-5.3 card 2026-10-14; read 2026-10-19 ✓xmodelOpus 4.8·max · Fable 5·max
Terminal-Bench 4.0
tbench
tbench.ai primary payload · 66 tasks × 330 trials, Claude Code, all at max; cost basis undocumented ✓xmodelOpus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5.1·xhigh · Fable 5.1·high · Fable 5.1·medium · Opus 5·xhigh · Opus 5·max · Opus 5·high · Opus 5·medium · Fable 5·max · Fable 5.1·low · Opus 5·low · Opus 4.8·max · Sonnet 5·max
Terminal-Bench 4.0 (OpenAI GPT-6 Astra post chart, non-code mode)
openai
Terminal-Bench 4.0; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5·max · Opus 5·max
Terminal-Bench 4.0 (xAI Grok 4.7 post)
xai
Terminal-Bench 4.0 'multi-hour terminal work' %, harness not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓xmodelFable 5.1·max
Terminal-Bench Science 0.1 (OpenAI GPT-6 Astra post chart, non-code mode)
openai
Terminal-Bench Science 0.1; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 5·max
Terminal-Bench Science 0.1 (OpenAI GPT-6.1 Sol post)
openai
Terminal-Bench Science 0.1; OpenAI GPT-6.1 Sol post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; Wayback first capture; read 2026-10-19 ✓xmodelOpus 5.5·max
Terminal-Bench v3.0 (xAI Grok 4.6 post)
xai
Terminal-Bench v3.0 %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓xmodelFable 5·max
Test Cabinet · claude 2.1.203 · shatter v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.203; case shatter v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelHaiku 4.5·nothink · Opus 4.8·nothink
Test Cabinet · claude 2.1.204 · shatter v1.0.0 · warhead
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.204; case shatter v1.0.0; variant warhead; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Haiku 4.5·nothink
Test Cabinet · claude 2.1.205 · fathom v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.205; case fathom v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Sonnet 4.6·nothink · Haiku 4.5·nothink
Test Cabinet · claude 2.1.206 · floe-bear v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.206; case floe-bear v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Haiku 4.5·nothink
Test Cabinet · claude 2.1.208 · fathom v1.1.0 · kindle
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.208; case fathom v1.1.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Haiku 4.5·nothink
Test Cabinet · claude 2.1.211 · valence v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.211; case valence v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Haiku 4.5·nothink
Test Cabinet · claude 2.1.233 · deepcore v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.233; case deepcore v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Opus 5·high
Test Cabinet · claude 2.1.233 · fathom v2.0.0 · kindle
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.233; case fathom v2.0.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Opus 5·high
Test Cabinet · claude 2.1.234 · arc-foundry v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case arc-foundry v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 5·high · Opus 4.8·nothink
Test Cabinet · claude 2.1.234 · deepcore v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case deepcore v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 5·high · Opus 4.8·nothink
Test Cabinet · claude 2.1.234 · pong v2.1.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case pong v2.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Opus 5·high
Test Cabinet · claude 2.1.235 · coil v1.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case coil v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelOpus 4.8·nothink · Opus 5·high
Test Cabinet · claude 2.1.235 · pong v2.1.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case pong v2.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelFable 5·high · Sonnet 4.6·nothink · Sonnet 5·high · Haiku 4.5·nothink · Opus 4.8·nothink
Test Cabinet · claude 2.1.235 · pong v2.1.0 · gyre
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case pong v2.1.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelFable 5·high · Opus 4.8·nothink · Opus 5·high
Test Cabinet · claude 2.1.237 · band-of-bots v1.0.0 · default
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.237; case band-of-bots v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelFable 5·high · Sonnet 5·high
Test Cabinet · claude 2.1.237 · pong v2.1.0 · gyre
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.237; case pong v2.1.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelSonnet 4.6·nothink · Opus 5·high
Test Cabinet · claude 2.1.238 · band-of-bots v1.0.0 · default
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.238; case band-of-bots v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelFable 5·high · Sonnet 5·high
Test Cabinet · claude 2.1.302 · pong v3.0.0 · base
testcabinet
snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.302; case pong v3.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓xmodelFable 5.1·high · Sonnet 5.5·high
tfcxzaqwerds: 9 hard reasoning questions x3 runs (no tools), score
note-tfcxzaqwerds
note.com/tfcxzaqwerds/n/nab4b132711a2; same Claude Code headless setup, effort high, no tools, maze/24-digit-multiplication/logic-puzzles, 9 questions x3 runs = 27 trials/model, 15-min time cap per question; read 2026-10-01 ✓xmodelSonnet 5·high · Sonnet 5.5·high
The Endless Exam (2609.24555) — 69 open-ended maths constructions, Claude Code tools off
arxiv-2609.24555
Table 2, 69 instances, overall score = 100 x mean relative quality vs reference; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelFable 5.1·medium · Fable 5.1·high · Opus 5.5·medium · Opus 5.5·high
Thinkingbox-bench pass@1 (507 tasks x20 trials, 5 stateful business domains)
arxiv-2608.19741
Thinkingbox-bench 507 tasks, N=20 trials, own MCP sandbox, simulated user GPT-5.4-mini, Azure-hosted API (local vLLM for Qwen), published 2026-08-19; cost = campaign total USD over 507x20 trials; read 2026-10-20 ✓xmodelOpus 5·medium
ThreadKeeper: summarise a 128k-token fake chat (facts, threads, inventions), composite v2
github-threadkeeper
neko-legends/threadkeeper results/leaderboard.csv; 128k corpus seed 12801; Claude Code 2.1.289; judge venice/gemini-3-8-flash (neutral family) for every arm; composite = 100*(0.45 facts + 0.30 threads + 0.25 invention_score); subscription, no cost; read 2026-10-15 ✓xmodelOpus 5.5·medium · Sonnet 5.5·medium · Opus 5.5·max
Tier-2 engineering solve items (of 25), Claude Code CLI no tools
arxiv-2610.06668
25 multi-regime engineering solve problems, text only, Claude Code CLI replacing the system prompt, effort high where exposed; Supp. Table 3 of arXiv 2610.06668; read 2026-10-15 ✓xmodelFable 5.1·high · Opus 5.5·high · Sonnet 5.5·high · Sonnet 5·high · Opus 4.8·high
Timeline-Bench (video editing), vendor-native harnesses (Codex CLI / Claude Code)
arxiv-2609.35143
Timeline-Bench: same 56 tasks, 1 run per task; harness = developer's own (GPT in Codex CLI 0.155.1, Claude in Claude Code 2.1.278); score = Res. %; cost = Table 7 mean model cost per run (Claude Code recorded; Codex CLI estimated by the authors from tokens at base-rate prices); curated-guidance and computer-use Astra variants not rows (different setups); read 2026-10-20 ✓xmodelOpus 5·max · Fable 5.1·max
Tinybird SQL benchmark
llm-benchmark.tinybird.live
Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓xmodelOpus 4.7·nothink · Fable 5.1·high · Opus 5.5·high · Sonnet 5.5·high · Opus 5·high · Sonnet 4.6·nothink · Haiku 5.5·nothink
TLA-Verus tlaverus_algo 40 tasks: p@3
tla-verus-2610.14923
TLA-Verus Table 12: tlaverus_algo 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓xmodelOpus 5·high
TLA-Verus tlaverus_llm 40 tasks: p@3
tla-verus-2610.14923
TLA-Verus Table 13: tlaverus_llm 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓xmodelOpus 5·high
TLA-Verus verusbench 150 tasks: p@3
tla-verus-2610.14923
TLA-Verus Table 11: verusbench 150 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓xmodelOpus 5·high
tokensave-brickbreaker-cost
tokensave
—xmodelHaiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·high
tokensave-brickbreaker-cost-first
tokensave
—xmodelSonnet 5.5·medium · Sonnet 5.5·high
tokensave-vibe-2048
tokensave
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium · Opus 5.5·medium
tokensave-vibe-brick
tokensave
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium · Opus 5.5·medium
tokensave-vibe-sudoku
tokensave
—xmodelHaiku 5.5·medium · Sonnet 5.5·medium · Opus 5.5·medium
Tool-Decathlon (GLM-5.2 card)
zhipu
Tool-Decathlon official evaluation service, max_token 128K; GLM-5.2 card; read 2026-10-19 ✓xmodelOpus 4.8·max
Toolathlon-Verified
anthropic
Sonnet 5.5 card table 8.14.5.A · 7 Claude models at max, Pass@1, score-only ✓xmodelSonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max
Toolathlon-Verified leaderboard (current, 108 tasks)
toolathlon
Toolathlon-Verified (released 30 Jun 2026), 108 tasks, 32 MCP servers / 604 tools + 7 local toolkits, Pass@1 % (+-CI) over 3 runs; rows carry the 'Evaluated by us' badge unless noted; Agent 'Default'; page fetched 2026-10-20 (newest row 2026-08-30); cells identical to held cells (vendor cards/blogs copying the board) dropped; read 2026-10-20 ✓xmodelOpus 4.8·max · Sonnet 5·max
Towards AI internal writing-Elo board, Sep 22, 2026
reddit-1wr03jc-img1
10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; top-16 leaderboard bar chart, footer 'Board of Sep 22, 2026'; values are printed bar labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓xmodelOpus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max
tuabench-cc
arxiv-2606.28480
—xmodelOpus 4.8·max · Opus 4.7·xhigh
tycho-arc3
arxiv-2607.28287
—xmodelOpus 4.8·xhigh · Opus 5·xhigh
UGI Leaderboard — UGI composite (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, fixed prompt set, composite UGI score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓xmodelSonnet 4.6·low · Sonnet 4.6·high · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high
UGI Leaderboard — World Model factual recall (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, recipe/geoguessr/weight/music percent-error battery; test dates vary by model; read 2026-10-05 ✓xmodelSonnet 4.6·low · Sonnet 4.6·high · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high
UGI Leaderboard — Writing quality (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, fixed prompt set, LLM-judged Writing score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓xmodelSonnet 4.6·low · Sonnet 4.6·high · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high
Vals AIME
vals.ai
Vals AI · AIME, compute_effort null (effort unverified), board 2026-04-16 ✓xmodelOpus 4.7·max
Vals CaseLaw v2
vals.ai
Vals AI · CaseLaw v2, compute_effort null (effort unverified), board 2026-05-04 ✓xmodelOpus 4.7·max
Vals Code Migration
vals.ai
Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓xmodelFable 5·max · Fable 5.1·max · Opus 4.7·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max
Vals CorpFin v2
vals.ai
Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelFable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Opus 5·max
Vals CUA-bench
vals.ai
Vals AI · CUA-bench via Claude Code, Opus 5.5, Opus 5 and Fable 5.1 at max — snapshot 27 Sep, re-priced (board 22 Sep) ✓xmodelFable 5.1·max · Opus 5·max · Opus 5.5·max
Vals CyberBench
vals.ai
Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓xmodelOpus 5·max · Fable 5.1·max · Sonnet 5·max · Sonnet 5.5·max · Opus 5.5·max · Haiku 5.5·max
Vals Finance Agent v2
vals.ai
Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·high · Sonnet 5·max · Sonnet 5.5·max
Vals GPQA
vals.ai
Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelFable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Opus 5·max
Vals HLAB
vals.ai
Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓xmodelFable 5·max · Fable 5.1·max · Opus 4.7·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max
Vals Index (29 Sep)
vals.ai
vals.ai Vals Index as recomputed on 29 Sep 2026 (older models dropped), max effort, accuracy and cost per test ✓xmodelFable 5.1·max · Opus 5·max · Fable 5·max · Opus 5.5·max · Sonnet 5.5·max · Opus 4.8·max · Sonnet 5·max · Haiku 5.5·max
Vals IOI
vals.ai
Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max
Vals Legal Research
vals.ai
Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5.2·high
Vals LegalBench
vals.ai
Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelOpus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max
Vals LiveCodeBench
vals.ai
Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓xmodelOpus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max
Vals MedCode
vals.ai
Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max
Vals MedScribe
vals.ai
Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max
Vals MMLU-Pro
vals.ai
Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelOpus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max
Vals MMMU
vals.ai
Vals AI · MMMU, Claude models at their stated compute_effort, measured cost/test ✓xmodelOpus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max
Vals MortgageTax
vals.ai
Vals AI · MortgageTax, Claude models at their stated compute_effort, measured cost/test ✓xmodelFable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Opus 5·max
Vals MysteryMechanism
vals.ai
Vals AI · MysteryMechanism, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5.5·max · Haiku 5.5·max
Vals ProgramBench
vals.ai
Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓xmodelFable 5·max · Fable 5.1·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max · Sonnet 5·max · Haiku 5.5·max · Sonnet 5.5·max
Vals ProofBench
vals.ai
Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓xmodelFable 5·max · Fable 5.1·max · Opus 5·max · Opus 5.5·max · Sonnet 5·max · Haiku 5.5·max
Vals Public Benefits v1
vals.ai
Vals AI · Public Benefits Bench v1 (board 2026-06-09), max + Haiku default ✓xmodelFable 5·max · Opus 4.8·max
Vals Public Benefits v1.1
vals.ai
Vals AI · Public Benefits v1.1, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max
Vals SAGE
vals.ai
Vals AI · grading student math work against a rubric, all at max ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max
Vals Science Index (biology data investigation, mechanism discovery, research workflows)
vals.ai
vals.ai Science Index, equal-weighted BioMysteryBench / MysteryMechanism / Terminal-Bench-Science; cost per test; read 2026-10-18 ✓xmodelOpus 5·max
Vals SRE-Bench
vals.ai
Vals AI · binary reverse engineering (despite the name), all at max — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5.5·max · Haiku 5.5·max · Sonnet 5·max
Vals SWE-bench
vals.ai
Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓xmodelFable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Opus 5·max
Vals Tax Agent
vals.ai
Vals AI · Tax Agent, 10 Claude entries at their stated compute_effort — snapshot 27 Sep, re-priced ✓xmodelOpus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5·max · Sonnet 5.5·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Haiku 5.5·max · Fable 5.2·high
Vals TaxEval v2
vals.ai
Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelFable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Opus 5·max
Vals TB 2.1
vals.ai
Vals AI · Terminal-Bench 2.1, Claude models at their stated compute_effort (Sonnet 5.5, Opus 5.5 high) — snapshot 27 Sep, re-priced ✓xmodelOpus 4.8·max
Vals Time Horizon KSP
vals.ai
Vals AI · Time Horizon Index: KSP, 3 Claude at max, board 2026-09-14 ✓xmodelFable 5.1·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max
vals.ai Robo-bench (robot arm + humanoid control in a physics sim, most tasks held out)
vals.ai
claude-code harness; overall score; cost = vals 'Est. API cost' per test (USD, estimate); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max
vals.ai SWE-bench Multimodal (JS front-end issues with screenshots)
vals.ai
Vals-run agent; resolve rate overall, cost per test (USD); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓xmodelOpus 5.5·max · Fable 5.1·max · Sonnet 5.5·max · Haiku 5.5·max
valscodemig11
vals.ai
—xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.2·high
valsemb11
vals.ai
—xmodelFable 5.1·max · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.2·high
valsfab21
vals.ai
—xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.2·high
valsfinindex
vals.ai
—xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.2·high
valshlab-2026-10-21
vals.ai
—xmodelOpus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max
valsindex4
vals.ai
—xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.2·high
valslegalindex
vals.ai
—xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.2·high
valssage11
vals.ai
—xmodelHaiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max
valstaxwp
vals.ai
—xmodelHaiku 5.5·max · Fable 5.2·high
valswsindex
vals.ai
—xmodelFable 5·max
Vectorise doc-search, judge-graded (Vidali gist)
vidali-gist
citation recall by cross-model judge, 22 Sep 2026 evening run ✓xmodelOpus 5·high · Sonnet 5·high
Vending-Bench 2 (Andon Labs)
andonlabs-vb2
andonlabs.com/evals/vending-bench-2: mean final money balance after a simulated year (start $500), mean of runs; 3 rows read from the page, the rest relayed by Epoch's mirror; no effort stated by the page; no cost; read 2026-10-15 ✓xmodelOpus 4.8·high · Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Opus 4.8·max
VeriContest-Lean (theorems proved of 10; cost per proved theorem)
arxiv-2610.03994
10 theorems x 5 Claude models x 5 effort settings, one attempt each, Claude Code, 10 iterations/3 h; Table 4-5 of arXiv 2610.03994; read 2026-10-15 ✓xmodelFable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max
vibe-openscad iter-png-1
github-vibeopenscad
vibe-openscad · 7 tasks, 1 PNG-feedback round, 5 Claude models ✓xmodelHaiku 4.5·nothink · Opus 4.8·nothink · Sonnet 4.6·nothink
vibe-openscad iter-png-2
github-vibeopenscad
vibe-openscad · 7 tasks, 2 PNG rounds ✓xmodelHaiku 4.5·nothink · Opus 4.8·nothink · Sonnet 4.6·nothink
vibe-openscad iter-png-3
github-vibeopenscad
vibe-openscad · 7 tasks, 3 PNG rounds ✓xmodelHaiku 4.5·nothink · Opus 4.8·nothink · Sonnet 4.6·nothink
vibe-openscad PDF page
github-vibeopenscad
vibe-openscad · 2 tier-4 datasheet tasks, 8 Claude models default ✓xmodelFable 5·high · Fable 5.1·high · Haiku 4.5·nothink · Opus 4.7·nothink · Opus 4.8·nothink · Opus 5·high · Sonnet 4.6·nothink · Sonnet 5·high
Video2World (222 instances), coding agents
arxiv-2610.04432
Video2World (arXiv 2610.04432, 3 Oct 2026): 222 instances (189 clips, 39 task families), agent builds a simulation package from video; score = V2WScore (0-100, first column of main table; build rate, functionality success, progress, geometry errors are other columns); seven models in OpenCode (via OpenRouter), GPT-6 Astra in Codex CLI, Fable-5.1 in Claude Code; cost = Table A17 mean API cost per episode (USD); read 2026-10-20 ✓xmodelFable 5.1·max
vlm-exam counting
github-vlmexam
roboflow/vlm-exam · 74 samples, LLM-judge accuracy ✓xmodelFable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high
vlm-exam detection
github-vlmexam
roboflow/vlm-exam · 250 images, mAP@50 ✓xmodelFable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high
vlm-exam extraction
github-vlmexam
roboflow/vlm-exam · 97 samples, LLM-judge accuracy; same models as OCR ✓xmodelFable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high
vlm-exam identification
github-vlmexam
roboflow/vlm-exam · 32 samples, near ceiling ✓xmodelFable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high
vlm-exam OCR
github-vlmexam
roboflow/vlm-exam · 37 samples, similarity; Fable 5.1 & Opus 5.5 low/high ×3 runs, others low ×1 ✓xmodelFable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high
vlmexam-text
github-vlmexam
—xmodelFable 5·low · Fable 5·high · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Sonnet 5·high · Opus 4.8·low · Opus 4.8·high · Opus 5·low · Opus 5·high · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high
vlnr2r
arxiv-2607.26148
—xmodelOpus 5·high · Sonnet 5·max · Opus 4.8·max · Fable 5·max · Opus 5·max
vulcanbench3
github-vulcanbench
—xmodelFable 5·low · Fable 5·high · Fable 5·medium
WeirdML v3 (agentic, token-axis area score)
epoch-archive
11 ML tasks x 15 configurations; score = token-axis area 0-1 with 95% CI; cost = mean API $/run; agent claude_code; read 2026-10-01 ✓xmodelOpus 5.5·xhigh · Fable 5.1·xhigh · Sonnet 5.5·xhigh · Opus 5·xhigh · Fable 5·xhigh
WhiteKUMALabo F5/O4.8 T2
whitekumalabo
Claude Code 2.1.170 · intro-text, blind 15pt judge ✓xmodelFable 5·high · Opus 4.8·high
WhiteKUMALabo F5/O4.8 T3
whitekumalabo
Claude Code 2.1.170 · task decomposition, blind 15pt judge ✓xmodelFable 5·high · Opus 4.8·high
WhiteKUMALabo F5/O4.8 T4
whitekumalabo
Claude Code 2.1.170 · 1-sentence summary, cost/speed only ✓xmodelFable 5·high · Opus 4.8·high
wmedia-php-hidden-cases
wmedia.es
—xmodelOpus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh
yama3133 · 400k-character log
qiita-yama3133
qiita 29 Sep · Claude Code -p minimal setup, 400k-character log, needle arithmetic; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓xmodelOpus 5.5·medium · Opus 5·medium · Sonnet 5·medium
yama3133 · 8 closed-answer problems × 3
qiita-yama3133
qiita 29 Sep · Claude Code -p minimal setup, 8 closed-answer problems × 3; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓xmodelOpus 5.5·medium · Opus 5·medium · Sonnet 5·medium
yama3133 · agentic bug fix
qiita-yama3133
qiita 29 Sep · Claude Code -p minimal setup, agentic bug fix, 5 planted bugs, 21 tests; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓xmodelOpus 5.5·medium · Opus 5·medium · Sonnet 5·medium
youkiti-prisma-tsuge10-accuracy
github-youkiti
—xmodelOpus 4.7·low
zapierab-domain-finance
zapier
—xmodelFable 5.2·max
zapierab-domain-hr
zapier
—xmodelFable 5.2·max
zapierab-domain-marketing
zapier
—xmodelFable 5.2·xhigh
τ^τ-Bench
arxiv-2609.04611
arXiv 2609.04611 · 53 agent-construction tasks in Claude Code, Opus 5 vs Sonnet 5 both at max, build spend at list ✓xmodelOpus 5·max · Sonnet 5·max
How the numbers are built — the method

Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.

In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.

The procedure

  1. Every number traced to its source. Each source is recorded as facts: the value as the source writes it, the exact text or table cell it is read from, the URL and the date of the capture; agents read every value line by line, and a measurement whose facts lack their proof stays out. Registries of models (the names each was served under, its effort settings and defaults, dated), routes (makers' APIs, resellers, routers, harnesses), benchmarks and their versions, publishers and dated prices let one build decide, by a fixed decision tree, which model, configuration and price each measurement belongs to.
  2. Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale, one version of a board; a group that mixes them is split. Every run is dated, and its model and effort are read as the maker and every route in between served them on that date — never assumed: a route the source does not name is never presumed to be the maker's API, and every route that could have served the run must give the same configuration. At the least doubt on the model, the effort or what was run, the measurement stays in the data file and out of the fit.
  3. Costs at today's prices. Every cost is put on the model's current price from the prices in force on its run date: the source's own figure at that price, the price change applied to the run's own token counts, or, when the token mix is unknown, a price band whose width is added to the cost's precision. A cost that cannot be brought to the current price stays out.
  4. Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The scale is read from the metric's label, never from the benchmark's identity; whether lower is better is read from the benchmark's registry, and such a score is turned in its own unit first (an attack success rate of 30 % reads 70 %); a score whose direction is not established stays out. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
  5. One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result. The sizes of these effects are estimated for each model maker on its own, not tied to the other makers'; a maker's own charts about its own models carry an estimated home effect, also each maker's own, left out of every value. The fit takes the couples of eight makers measured in the same groups (Anthropic, OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi), on one scale; this page shows the Claude couples. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
  6. Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source; the one weight set by rule is that of a single run: when a source gives k quality measurements of one run (several metrics, sub-scores or slices of the same executions), each weighs 1/k, so the run counts as one measurement.
  7. Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, a board copying another's numbers, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. Versions of one board that re-grade the same runs (recognised by identical costs) count those runs once. Before the fit, two groups that still share identical costs on three couples or more stop it until the copy is resolved or the coincidence proven from the sources. An index keeps, for a couple whose index components are already in the data, only what its missing components add, or is left out.
  8. Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
  9. From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.

Key values

  • Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
  • Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
  • Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit five figures on this page can move by one unit in that digit (the cost of Fable 5.1 at max, the cost of Fable 5.2 at max, the cost of Fable 5 at max, the cost of Opus 5.5 at max, the quality of Opus 4.7 at nothink), far inside their intervals.
  • Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.

Techniques and software

  • Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes, six chains whose draws continue in the same run until the criteria are met (CmdStan only to resume a stopped fit from its stored draws, or to restart a chain left in a far region from a good chain's last draw); built and checked stage by stage following the Bayesian workflow.
  • Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
  • Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
  • Scales: the empirical logit of bounded scores, with the binomial shape of its noise.

Checks (on the fit of 29 September 2026, before the other makers' couples joined the data). Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.

Alexandre Gensse · github.com/alexandregensse-blip

All figures are indicative and for informational purposes only — derived from public third-party measurements, not an official benchmark, and not affiliated with or endorsed by any model maker. Verify before relying on any value.

Data and text under CC BY 4.0 · code under MIT · download the data (CSV)