Qwen models compared cost vs quality at every effort level

Data analysis · aggregated from 47 sources · 188 benchmarks · 412 measurements
GitHub
Alexandre Gensse

What each recent Qwen model actually costs, at every effort level, and the quality it delivers — fused from public benchmark measurements for every (model, effort) pair, on the same scales as every other maker's on this site.

The best value today is Qwen3.8 Flash with thinking off — 7.9× cheaper than the price trend for its quality (expected score 43.5 % on the benchmark panel).

Quality is the expected score across the whole benchmark panel, not one benchmark: the type of task can move it, see what moves everything.

Measurements
47 sources · 188 benchmarks · 412 measurements
Updated
19 Nov 2026
👑 Best overall
Qwen3.8 Flash · Thinking off Cost 1.3× · Score 43.5 %
787
Grunt work
Qwen3.7 Flash · Thinking off Cost 1.0× · Score 40.5 %
610
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Everyday tasks
Qwen3.8 Flash · Thinking off Cost 1.3× · Score 43.5 %
787
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Advanced reasoning
Qwen3.8 Max 0902 build · Medium Cost 79× · Score 64.7 %
212
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Cutting-Edge thinking
Qwen3.8 Max 0902 build · Medium Cost 79× · Score 64.7 %
212
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.

Quality vs. cost of every Qwen model, at each effort level

One point per couple: a model at one configuration (thinking off, minimal, low → max, thinking on); no line joins a model's couples. Only the couples of the Pareto frontier are labelled (model · effort), where there is room; hover or tap any point for its model, effort, cost and score. Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward. A model's builds and modes — a dated build, a preview, a Pro mode — share its colour: the first is drawn with filled points, the others with hollow points, and its label and tooltip say what it is. They are separate couples, each with its own effort ladder.

Best Qwen model for the price — the Pareto frontier

The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated; the frontier couples are labelled (model · effort) where there is room — hover or tap any point for its model, effort, cost and score. No line joins a model's couples. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.

Value — distance to the price trend

The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 9 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.64, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.

Model · effortcostexpected scorevalue index
trend = 100
Qwen3.8 Flash · Thinking off1.3×43.5 %787
Qwen3.7 Flash · Thinking off1.0×40.5 %610
Qwen3.8 Max 0902 build · Medium79×64.7 %212
Qwen3.8 Max · Low78×61.2 %137
Qwen3.8 Max 0902 build · Low77×58.8 %102
Qwen3.8 Max · xHigh251×66.7 %87
Qwen3.8-27B · Medium39×52.0 %85
Qwen3.8-27B · Low30×49.1 %74

Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.

Best value by task tier — what to actually run

Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.

Target 41.6 % – Grunt work
Qwen3.7 Flash · Thinking off Cost 1.0× · Score 40.5 %
610
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Target 48.7 % – Everyday tasks
Qwen3.8 Flash · Thinking off Cost 1.3× · Score 43.5 %
787
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Target 56.9 % – Advanced reasoning
Qwen3.8 Max 0902 build · Medium Cost 79× · Score 64.7 %
212
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Target 65.4 % – Cutting-Edge thinking
Qwen3.8 Max 0902 build · Medium Cost 79× · Score 64.7 %
212
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.
👑 Best overall
Qwen3.8 Flash · Thinking off
Cost 1.3× · Score 43.5 % · Value index 787

Picked as the frontier couple that sits furthest below the price trend : it costs 7.9× less than the trend charges for its quality — or, read on the other axis, it scores 11.1 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.

Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — Qwen effort levels compared

Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.

Modelthinking offlowmediumhighxhighmaxExpected score
top effort
Qwen3.8 Max—
7854–111
——
251184–341
—66.7 %
Qwen3.8 Max 0902 build—
7757–106
7955–111
—
10375–142
—62.5 %
Qwen3.8-27B—
3024–38
3926–59
—
8055–118
—50.1 %
Qwen3.8 Flash
1.30.95–1.7
—————43.5 %
Qwen3.7 Flash
1.00.74–1.4
—————40.5 %

"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. A model measured at a single setting (solo : no effort parameter, see the measurement sources) fills one merged cell. A thinking off column appears when a model is measured with its thinking switched off, a minimal one for the lowest level some models have below low; each is a configuration of its own, read from its own measurements. n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).

Value-index matrix — every model and effort against the price trend

The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.

Modelthinking offlowmediumhighxhighmax
Qwen3.8 Max—
13782–229
——
8756–137
—
Qwen3.8 Max 0902 build—
10267–156
212103–435
—
12380–189
—
Qwen3.8-27B—
7451–108
8553–137
—
3220–50
—
Qwen3.8 Flash
787420–1477
—————
Qwen3.7 Flash
610395–943
—————

Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.

What moves everything — four factors behind the numbers

The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.

Task type

Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.

Complexity

The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).

Generation

Each release buys the same quality for less: turn on Older models to set each family's earlier models beside its current one, on the same scales.

Cache & harness

Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.

Data Sources

Every source below measured at least two (model, effort) couples on the same task; 10115 cost and 23548 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.

SourceActual config (harness · effort)TypeLinked couples (same task)
AA CritPt
artificialanalysis
AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
AA GDP-PDF
artificialanalysis
AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink
AA GDPval v2
artificialanalysis
AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓sweepQwen3.8-27B·medium · Qwen3.8-27B·low · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
AA HLE
artificialanalysis
AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
AA Index
artificialanalysis
AA-Index · cross-model at max + Opus 5 sweep low→max ✓sweepQwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink
AA MLCR
artificialanalysis
AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓sweepQwen3.8-27B·xhigh
AA Omniscience
artificialanalysis
AA model payload · Omniscience accuracy (not the penalised index) ✓sweepQwen3.8-27B·medium · Qwen3.8-27B·low · Qwen3.5-122B-A10B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink
AA SciCode
artificialanalysis
AA model payload · SciCode, 4 models × ladder + 3 at max ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink
AA TB 4.0
artificialanalysis
AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink
AA Terminal-Bench-Science
artificialanalysis
AA model payload · Terminal-Bench-Science 0.1 (AA harness, separate from Vals's board), score-only, 4 models × ladder + 2 at max ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium
AA-Briefcase
artificialanalysis
AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink
AA-LCR
artificialanalysis
AA model payload · long-context reasoning, input-dominated cost ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
ARC-AGI-1
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·medium · Qwen3.8-27B·low
ARC-AGI-2
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓sweepQwen3.8-27B·xhigh · Qwen3.8-27B·medium · Qwen3.8-27B·low
BullshitBench v2
github-bullshitbench
petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓sweepQwen3.8 Max·xhigh · Qwen3.8-27B·xhigh
ibragim.dev · Pi
ibragim-dev
Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Pi 0.85.1, effort per row, pass@1 and $/task; board of 28 Sep ✓sweepQwen3.8-27B·xhigh
ObviousBench 0.2
benchmarklist
inspect-ai · 144 items ×3 epochs, 3 models × full ladder, measured tokens ✓sweepQwen3.7 Flash·nothink · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Qwen3.8 Flash·nothink · Qwen3.8 Max·low · Qwen3.8 Max·xhigh · Qwen3.8 Max 0902 build·low · Qwen3.8 Max 0902 build·medium · Qwen3.8 Max 0902 build·xhigh
QuoteBench
arxiv-2608.13547
arXiv 2608.13547 · 56 frozen tasks, provider-reported output tokens, 3 models × ladder ✓sweepQwen3.5-27B·nothink
vibe-openscad
github-vibeopenscad
output_config.effort explicit · 3 models × ladder; quality saturates → cost only ✓sweepQwen3.8-27B·xhigh
vlm-exam
github-vlmexam
roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓sweepQwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink
Official tau2-bench leaderboard, banking_knowledge (AllTools), Claude submissions by Sierra
github-sierra-tau2-bench
banking_knowledge domain, 4 trials, user simulator gpt-5.2 (low), reasoning_effort max for all; pass^1 %; read 2026-10-18 ✓xgenQwen3.8 Max·xhigh
AA /task v4
artificialanalysis
AA launch article · per-task $, Fable 5.1 xhigh/max vs Fable 5/Opus 5 max ✓xmodelQwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.8 Max·xhigh
AA AutomationB.
artificialanalysis
AA evaluation page · 657 tasks partial-credit, 3 models at max ✓xmodelQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink
AA FrontierCode 1.1 (AA-run, weighted score)
artificialanalysis
Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelQwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium
AA MMMU-Pro
artificialanalysis
Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelQwen3.5-27B·nothink · Qwen3.6-27B·nothink · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
aa-index5
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low
aa5-aa-briefcase
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low
aa5-critpt-v2-hard
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low
aa5-frontiercode-2-0
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low
aa5-hle-diamond
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low
aa5-mlcr-aa
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low
aa5-omniscience-non-hallucination
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low
aagpqa
artificialanalysis
—xmodelQwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.8 Max·xhigh
aaifbench
artificialanalysis
—xmodelQwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
aaomni-index
artificialanalysis
—xmodelQwen3.8-27B·medium · Qwen3.8-27B·low · Qwen3.5-122B-A10B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink
aatau-banking
artificialanalysis
—xmodelQwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium
aatau2
artificialanalysis
—xmodelQwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
aatb21
artificialanalysis
—xmodelQwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.8 Max·xhigh
aatbhard
artificialanalysis
—xmodelQwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink
Agents' Last Exam ALE-CLI (GLM-5.3 card)
zhipu
ALE official protocol, Claude Code harness, 105 tasks, effort=max 1M context 64K output, 4h timeout; GLM-5.3 card; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high)
aibenchy
aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓xmodelQwen3.8 Max 0902 build·medium · Qwen3.8 Max·low · Qwen3.8 Max 0902 build·low · Qwen3.5-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.6-27B·nothink · Qwen3.7 Flash·nothink · Qwen3.5-35B-A3B·nothink · Qwen3-Coder-Next·nothink
arxiv-2609.29933-t1-longdocurl-agent-6-tool-fc
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t1-mmlongbench-doc-agent-6-tool-fc
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t1-mmlongbench-doc-all-images
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-longdocurl-ldurl-column-k-10
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-longdocurl-ldurl-column-k-2
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-longdocurl-ldurl-column-k-20
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-longdocurl-ldurl-column-k-5
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-10
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-2
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-20
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-5
arxiv-2609.29933
—xmodelQwen3.5-27B·nothink
arxiv-2610.07563-householdbench-avg-rank
arxiv-2610.07563
—xmodelQwen3.6-27B·nothink
arxiv-2610.12596-spirebench-acts
arxiv-2610.12596
—xmodelQwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh
arxiv-2610.13075-longbench-v2-direct
arxiv-2610.13075
—xmodelQwen3.8-27B·xhigh
arxiv-2610.13075-officeqa-v1-direct
arxiv-2610.13075
—xmodelQwen3.8-27B·xhigh
arxiv-2610.13075-officeqa-v2-direct
arxiv-2610.13075
—xmodelQwen3.8-27B·xhigh
arxiv-2610.17016-covert-kl-nats
arxiv-2610.17016
—xmodelQwen3.8 Flash·nothink
arxiv-2610.17016-credit-kl-nats
arxiv-2610.17016
—xmodelQwen3.8 Flash·nothink
arxiv-2610.17016-peer-kl-nats
arxiv-2610.17016
—xmodelQwen3.8 Flash·nothink
arxiv-2610.30116-author-difficulty
arxiv-2610.30116
—xmodelQwen3.8-27B·xhigh
arxiv-2610.30116-author-hintlift
arxiv-2610.30116
—xmodelQwen3.8-27B·xhigh
arxiv-2610.30116-solve
arxiv-2610.30116
—xmodelQwen3.8-27B·xhigh
arxiv-2610.38249-data
arxiv-2610.38249
—xmodelQwen3.5-27B·nothink
arxiv-2610.38249-knowledge
arxiv-2610.38249
—xmodelQwen3.5-27B·nothink
arxiv-2610.38249-mean
arxiv-2610.38249
—xmodelQwen3.5-27B·nothink
arxiv-2610.38249-workout
arxiv-2610.38249
—xmodelQwen3.5-27B·nothink
arxiv-2611.02075-fewshot
arxiv-2611.02075
—xmodelQwen3.5-27B·nothink
arxiv-2611.02075-zeroshot
arxiv-2611.02075
—xmodelQwen3.5-27B·nothink
blog-qwen3.8: $OneMillion-Bench (expert score)
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark '$OneMillion-Bench (expert score)'; no cost or tokens in the source (score only); footnotes: "Evaluated using gemini-3.1-pro-preview."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: Agents' Last Exam (Pass / Score) [pass]
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Agents' Last Exam (Pass / Score)' [pass]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: AndroidBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'AndroidBench'; no cost or tokens in the source (score only); footnotes: "Evaluated on the 95-task public subset, reporting avg@3 scores."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: Automation-Bench (Pass@1)
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Automation-Bench (Pass@1)'; no cost or tokens in the source (score only); footnotes: "Evaluated on the 600-task public subset."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: CoWorkBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'CoWorkBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: DeepSWE 1.1
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'DeepSWE 1.1'; no cost or tokens in the source (score only); footnotes: "We report the highest score among both harnesses"; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: GPQA Diamond
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'GPQA Diamond'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: HealthBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HealthBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: HLE
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HLE'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: HLE w/ tools
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HLE w/ tools'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: IFBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'IFBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: JobBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'JobBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: LongBench v2
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'LongBench v2'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: MLS-Bench-Lite
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'MLS-Bench-Lite'; no cost or tokens in the source (score only); footnotes: "All other model scores are taken from the official leaderboard." / "Evaluated with Claude Code using a 5-hour timeout and max_tokens=131,072."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: MRCR v2 256K (8-needle)
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'MRCR v2 256K (8-needle)'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: PaperBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PaperBench'; no cost or tokens in the source (score only); footnotes: "Evaluated in the BasicAgent setting under Code-Dev mode, judged by Claude Opus 4.6, and averaged over 3 runs (max 12 hours per run)."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: PLawBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PLawBench'; no cost or tokens in the source (score only); footnotes: "Evaluated using gemini-3.1-pro-preview."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: PRBench-Finance
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PRBench-Finance'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: PRBench-Legal
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PRBench-Legal'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: QwenQoderBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenQoderBench'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness. Reporting avg@5 with a 6-hour timeout"; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: QwenReactBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenReactBench'; no cost or tokens in the source (score only); footnotes: "Inhouse React project building benchmark using Claude Code as the harness"; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: QwenSVGBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenSVGBench'; no cost or tokens in the source (score only); footnotes: "Inhouse SVG code generation benchmark; bilingual (EN/CN), auto-render + multimodal judge; BT/Elo rating."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: QwenSWEBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenSWEBench'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness. Reporting avg@3 with an 8-hour timeout"; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: SkillsBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'SkillsBench'; no cost or tokens in the source (score only); footnotes: "Opus 4.8 and Fable 5 are evaluated on Claude Code" / "GPT-5.6 Sol is evaluated on Codex" / "the Qwen-series are evaluated on OpenCode. All results are from our own testing."; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: Toolathlon Verified (Pass@1)
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Toolathlon Verified (Pass@1)'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
blog-qwen3.8: WorkSpaceBench
alibaba
launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'WorkSpaceBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓xmodelQwen3.8 Max·xhigh
bsb-v1-green
github-bullshitbench-v1
—xmodelQwen3.8 Max·xhigh · Qwen3.8-27B·xhigh
BVB
arxiv-2609.15478
arXiv 2609.15478 · Mini-BVB via LiteLLM, $3/scene cap, 5 Claude models at high (+ omitted-effort runs), LiteLLM list cost ✓xmodelQwen3.5-27B·nothink
Conceptual Reasoning Index · DTBench
conceptualreasoning.ai
Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓xmodelQwen3.8-27B·xhigh
Conceptual Reasoning Index · LMCA
conceptualreasoning.ai
Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓xmodelQwen3.8-27B·xhigh
contextarena-needle8-auc128k
contextarena.ai
—xmodelQwen3.8 Max·low · Qwen3.8 Max·xhigh
ctala
github-ctala
Claude Code CLI · reasoning not configured, temp 0.7 ✓xmodelQwen3.8 Max·xhigh
DataLLM Lab: 9 executed Python tasks, Qwen3.8 Max family (Prime, Max, Max-0902), pass and cost per 1,000 tasks
datallmlab
9 executed Python tasks (python3 -I subprocess, 12 s timeout), temperature 0, max_tokens 4000, one scored attempt, via OpenRouter; three runs on different days (Prime 2026-10-02, Max 2026-08-20, Max-0902 2026-09-16); cost = measured tokens x list price on the priced-on date (source-computed, 'not a billing statement'); read 2026-10-20 ✓xmodelQwen3.8 Max 0902 build·xhigh
EMB
vals.ai
Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Epoch · chess puzzles
epoch.ai
Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.7 Flash·nothink · Qwen3.8 Max 0902 build·xhigh
Epoch · frontiermath tier 4 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.8 Max 0902 build·xhigh
Epoch · frontiermath tiers 1 3 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.8 Max 0902 build·xhigh · Qwen3.7 Flash·nothink · Qwen3.6-27B·nothink
Epoch · furniture assembly
epoch.ai
Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.8 Max 0902 build·xhigh
Epoch · gpqa diamond
epoch.ai
Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.8 Max 0902 build·xhigh · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.7 Flash·nothink
Epoch · mystery game puzzles
epoch.ai
Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.6-27B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.7 Flash·nothink
Epoch · otis mock aime 2024 2025
epoch.ai
Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.7 Flash·nothink
Epoch · simpleqa verified
epoch.ai
Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelQwen3.8 Max 0902 build·xhigh
epoch-ext-surface-evolver-bench
epoch.ai
—xmodelQwen3.8-27B·xhigh
FlavourBench Score (mean of 3 families x 178 tasks)
flavourbench-2608.20574
FlavourBench Table 3 leaderboard, 534 tasks, FB Score = equal-family mean, 2 collection panels pooled, OpenRouter fixed provider (GLM 5.3 direct); arXiv 2608.20574 (20 Aug 2026); family columns Table 10 are same runs (rule 8); read 2026-10-20 ✓xmodelQwen3.8 Max·low
github-telecom-oam-objective-off
github-telecom-oam
—xmodelQwen3.8 Flash·nothink
github-telecom-oam-scenario-off
github-telecom-oam
—xmodelQwen3.8 Flash·nothink
HouseholdBench: household-economics prediction, macro-F1 on categorical targets
arxiv-2610.07563
32 household tasks, pre-cutoff test split, default reasoning effort; Table 3 of arXiv 2610.07563; read 2026-10-15 ✓xmodelQwen3.6-27B·nothink
ibmartian-full-p-other
ironbee
—xmodelQwen3.8 Max 0902 build·low · Qwen3.8 Max 0902 build·xhigh
IndustryBench Chinese, Final (SV), non-thinking vs thinking
arxiv-2605.10267
IndustryBench (arXiv 2605.10267 v3): 2,049 Chinese industrial-procurement QA items, zero-shot closed-book, raw 0-3 by Qwen3-Max judge minus safety-violation penalty (Final SV, 0-3); outputs collected Feb 2026; Table 7 = default non-reasoning run, Table 8 = same items/judge with thinking enabled; read 2026-10-19 ✓xmodelQwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink
llm2014-logic-2026-10-median
llm2014-logic
—xmodelQwen3.8-27B·xhigh
Mercor APEX-Agents
mercor.com
Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelQwen3.8-27B·xhigh
Mercor APEX-v1 (The AI Productivity Index)
mercor.com
Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelQwen3.8-27B·xhigh
mercor-apexv1-mean
mercor.com
—xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Case Government (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_case_marking@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Completion (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_cloze@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Famous Phrases (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_phrases@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Flora & Fauna (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_nature_translations@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Generative Quality (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark generative_quality@v1 primary metric quality_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Icelandic GEC (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_gec@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Icelandic WikiQA (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_wikiqa@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Meanings (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_meanings@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Song Lyrics (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_song_lyrics@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
Mideind Icelandic LLM leaderboard, Textile Terms (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_textiles@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelQwen3.8-27B·xhigh
NonoBench
nonobench.com
NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓xmodelQwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh
NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections)
matsonj/eval-connections
results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓xmodelQwen3.8-27B·low
OSWorld-Pro
arxiv-2609.24890
arXiv 2609.24890 · OSWorld harness, 250 turns, 4 Claude models all at max, tokens × OpenRouter list incl. cache ✓xmodelQwen3.8-27B·xhigh
patterneval-acc
arxiv-2608.12781
—xmodelQwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.6-27B·nothink
Roboflow Playground Vision Evals, Counting (low tier)
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Counting; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-counting) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓xmodelQwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink
Roboflow Playground Vision Evals, Identification (low tier)
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Identification; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-identification) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓xmodelQwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink
Roboflow Playground Vision Evals, Object Detection (low tier)
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Object Detection; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-detection) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓xmodelQwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink
Roboflow Playground Vision Evals, OCR (low tier)
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: OCR; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-ocr) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓xmodelQwen3.8 Flash·nothink · Qwen3.7 Flash·nothink
Roboflow Playground Vision Evals, overall (mean of tasks) with cost per sample
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; Overall = page's mean over the tasks scored (4/5 where OCR is missing); cost = 'Avg cost / sample' in USD from price x tokens at the pricing date 2026-10-19 ('~' = estimated from self-host/OpenRouter pricing); read 2026-10-20 ✓xmodelQwen3.8 Flash·nothink · Qwen3.7 Flash·nothink
Roboflow Playground Vision Evals, Reasoning (low tier)
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Reasoning; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓xmodelQwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink
SenseBench (WSD) · prompt p001
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelQwen3.8 Max 0902 build·xhigh · Qwen3.8 Max 0902 build·low · Qwen3.8-27B·low
SpireBench: Slay the Spire Ascension 20 Ironclad, mean final floor reached
arxiv-2610.12596
10 runs per model on the same held-out seeds, themed reskin with note-taking; Table 1 and Table 26 of arXiv 2610.12596; read 2026-10-15 ✓xmodelQwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh
SuperCLUE 2026年7月 幻觉控制
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelQwen3.8 Max 0902 build·xhigh
SuperCLUE 2026年7月 数学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelQwen3.8 Max 0902 build·xhigh
SuperCLUE 2026年7月 智能体任务规划
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体任务规划, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelQwen3.8 Max 0902 build·xhigh
SuperCLUE 2026年7月 智能体编程
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体编程, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelQwen3.8 Max 0902 build·xhigh
SuperCLUE 2026年7月 科学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelQwen3.8 Max 0902 build·xhigh
SuperCLUE 2026年7月 精确指令遵循
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelQwen3.8 Max 0902 build·xhigh
Surface Evolver bench
yhenon-surface-evolver
yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓xmodelQwen3.8-27B·xhigh
surfevolver-pass
yhenon-surface-evolver
—xmodelQwen3.8-27B·xhigh
synthbench-plain-globalopinionqa
github-synthbench
—xmodelQwen3.8 Flash·nothink
synthbench-plain-gss
github-synthbench
—xmodelQwen3.8 Flash·nothink
synthbench-plain-opinionsqa
github-synthbench
—xmodelQwen3.8 Flash·nothink
synthbench-plain-subpop
github-synthbench
—xmodelQwen3.8 Flash·nothink
Tinybird SQL benchmark
llm-benchmark.tinybird.live
Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓xmodelQwen3.8 Max·xhigh
UGI Leaderboard — UGI composite (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, fixed prompt set, composite UGI score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓xmodelQwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink
UGI Leaderboard — World Model factual recall (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, recipe/geoguessr/weight/music percent-error battery; test dates vary by model; read 2026-10-05 ✓xmodelQwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink
UGI Leaderboard — Writing quality (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, fixed prompt set, LLM-judged Writing score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓xmodelQwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink
Vals Code Migration
vals.ai
Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals CorpFin v2
vals.ai
Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelQwen3.8 Max·xhigh
Vals CyberBench
vals.ai
Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals Finance Agent v2
vals.ai
Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals GPQA
vals.ai
Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelQwen3.8-27B·xhigh · Qwen3.8 Max·xhigh
Vals HLAB
vals.ai
Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals IOI
vals.ai
Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals Legal Research
vals.ai
Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals LegalBench
vals.ai
Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals LiveCodeBench
vals.ai
Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓xmodelQwen3.8-27B·xhigh · Qwen3.8 Max·xhigh
Vals MedCode
vals.ai
Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓xmodelQwen3.8-27B·xhigh
Vals MedScribe
vals.ai
Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓xmodelQwen3.8-27B·xhigh
Vals MMLU-Pro
vals.ai
Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelQwen3.8-27B·xhigh · Qwen3.8 Max·xhigh
Vals MMMU
vals.ai
Vals AI · MMMU, Claude models at their stated compute_effort, measured cost/test ✓xmodelQwen3.8-27B·xhigh · Qwen3.8 Max·xhigh
Vals MortgageTax
vals.ai
Vals AI · MortgageTax, Claude models at their stated compute_effort, measured cost/test ✓xmodelQwen3.8-27B·xhigh · Qwen3.8 Max·xhigh
Vals ProgramBench
vals.ai
Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals ProofBench
vals.ai
Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓xmodelQwen3.8-27B·xhigh
Vals SAGE
vals.ai
Vals AI · grading student math work against a rubric, all at max ✓xmodelQwen3.8-27B·xhigh
Vals SWE-bench
vals.ai
Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓xmodelQwen3.8-27B·xhigh · Qwen3.8 Max·xhigh
Vals TaxEval v2
vals.ai
Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelQwen3.8-27B·xhigh · Qwen3.8 Max·xhigh
valstaxwp
vals.ai
—xmodelQwen3.8-27B·xhigh
vlm-exam counting
github-vlmexam
roboflow/vlm-exam · 74 samples, LLM-judge accuracy ✓xmodelQwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink
vlm-exam detection
github-vlmexam
roboflow/vlm-exam · 250 images, mAP@50 ✓xmodelQwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink
vlm-exam extraction
github-vlmexam
roboflow/vlm-exam · 97 samples, LLM-judge accuracy; same models as OCR ✓xmodelQwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink
vlm-exam identification
github-vlmexam
roboflow/vlm-exam · 32 samples, near ceiling ✓xmodelQwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink
vlm-exam OCR
github-vlmexam
roboflow/vlm-exam · 37 samples, similarity; Fable 5.1 & Opus 5.5 low/high ×3 runs, others low ×1 ✓xmodelQwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink
vlmexam-text
github-vlmexam
—xmodelQwen3.8 Max 0902 build·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink
zhenhuanbench-reasoning
arxiv-2610.26059
—xmodelQwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh
How the numbers are built — the method

Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.

In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.

The procedure

  1. Every number traced to its source. Each source is recorded as facts: the value as the source writes it, the exact text or table cell it is read from, the URL and the date of the capture; agents read every value line by line, and a measurement whose facts lack their proof stays out. Registries of models (the names each was served under, its effort settings and defaults, dated), routes (makers' APIs, resellers, routers, harnesses), benchmarks and their versions, publishers and dated prices let one build decide, by a fixed decision tree, which model, configuration and price each measurement belongs to.
  2. Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale, one version of a board; a group that mixes them is split. Every run is dated, and its model and effort are read as the maker and every route in between served them on that date — never assumed: a route the source does not name is never presumed to be the maker's API, and every route that could have served the run must give the same configuration. At the least doubt on the model, the effort or what was run, the measurement stays in the data file and out of the fit.
  3. Costs at today's prices. Every cost is put on the model's current price from the prices in force on its run date: the source's own figure at that price, the price change applied to the run's own token counts, or, when the token mix is unknown, a price band whose width is added to the cost's precision. A cost that cannot be brought to the current price stays out.
  4. Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The scale is read from the metric's label, never from the benchmark's identity; whether lower is better is read from the benchmark's registry, and such a score is turned in its own unit first (an attack success rate of 30 % reads 70 %); a score whose direction is not established stays out. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
  5. One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result. The sizes of these effects are estimated for each model maker on its own, not tied to the other makers'; a maker's own charts about its own models carry an estimated home effect, also each maker's own, left out of every value. The fit takes the couples of eight makers measured in the same groups (Anthropic, OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi), on one scale; this page shows the Qwen couples. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
  6. Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source; the one weight set by rule is that of a single run: when a source gives k quality measurements of one run (several metrics, sub-scores or slices of the same executions), each weighs 1/k, so the run counts as one measurement.
  7. Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, a board copying another's numbers, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. Versions of one board that re-grade the same runs (recognised by identical costs) count those runs once. Before the fit, two groups that still share identical costs on three couples or more stop it until the copy is resolved or the coincidence proven from the sources. An index keeps, for a couple whose index components are already in the data, only what its missing components add, or is left out.
  8. Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
  9. From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.

Key values

  • Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
  • Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
  • Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit three figures on this page can move by one unit in that digit (the cost of Qwen3-Coder-Next at nothink, the cost of Qwen3.8 Max at xhigh, the quality of Qwen3.8 Max 0902 build at medium), far inside their intervals.
  • Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.

Techniques and software

  • Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes, six chains whose draws continue in the same run until the criteria are met (CmdStan only to resume a stopped fit from its stored draws, or to restart a chain left in a far region from a good chain's last draw); built and checked stage by stage following the Bayesian workflow.
  • Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
  • Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
  • Scales: the empirical logit of bounded scores, with the binomial shape of its noise.

Checks (on the fit of 29 September 2026, before the other makers' couples joined the data). Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.

Alexandre Gensse · github.com/alexandregensse-blip

All figures are indicative and for informational purposes only — derived from public third-party measurements, not an official benchmark, and not affiliated with or endorsed by any model maker. Verify before relying on any value.

Data and text under CC BY 4.0 · code under MIT · download the data (CSV)