DeepSeek models compared cost vs quality at every effort level

Data analysis · aggregated from 82 sources · 403 benchmarks · 749 measurements
GitHub
Alexandre Gensse

What each recent DeepSeek model actually costs, at every effort level, and the quality it delivers — fused from public benchmark measurements for every (model, effort) pair, on the same scales as every other maker's on this site.

The best value today is DeepSeek V4.1 Flash at high effort — 1.2× cheaper than the price trend for its quality (expected score 57.9 % on the benchmark panel).

Quality is the expected score across the whole benchmark panel, not one benchmark: the type of task can move it, see what moves everything.

Measurements
82 sources · 403 benchmarks · 749 measurements
Updated
19 Nov 2026
👑 Best overall
DeepSeek V4.1 Flash · High Cost 2.3× · Score 57.9 %
121
Grunt work
DeepSeek V4.1 Flash 1017 build · Thinking off Cost 1.0× · Score 43.5 %
95
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Everyday tasks
DeepSeek V4.1 Flash · High Cost 2.3× · Score 57.9 %
121
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Advanced reasoning
DeepSeek V4.1 Flash 1017 build · Low Cost 2.9× · Score 61.3 %
119
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Cutting-Edge thinking
DeepSeek V4.1 Flash 1017 build · Max Cost 4.3× · Score 62.6 %
89
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.

Quality vs. cost of every DeepSeek model, at each effort level

One point per couple: a model at one configuration (thinking off, minimal, low → max, thinking on); no line joins a model's couples. Only the couples of the Pareto frontier are labelled (model · effort), where there is room; hover or tap any point for its model, effort, cost and score. Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward. A model's builds and modes — a dated build, a preview, a Pro mode — share its colour: the first is drawn with filled points, the others with hollow points, and its label and tooltip say what it is. They are separate couples, each with its own effort ladder.

Best DeepSeek model for the price — the Pareto frontier

The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated; the frontier couples are labelled (model · effort) where there is room — hover or tap any point for its model, effort, cost and score. No line joins a model's couples. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.

Value — distance to the price trend

The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 4 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.88, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.

Model · effortcostexpected scorevalue index
trend = 100
DeepSeek V4.1 Flash · High2.3×off-peak 1.1×57.9 %121
DeepSeek V4.1 Flash 1017 build · Low2.9×off-peak 1.5×61.3 %119
DeepSeek V4.1 Flash 1017 build · Thinking off1.0×off-peak 0.50×43.5 %95
DeepSeek V4.1 Flash 1017 build · Max4.3×off-peak 2.1×62.6 %89
DeepSeek V4.1 Flash · Max5.8×off-peak 2.9×63.3 %68

Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.

Best value by task tier — what to actually run

Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.

Target 44.4 % – Grunt work
DeepSeek V4.1 Flash 1017 build · Thinking off Cost 1.0× · Score 43.5 %
95
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Target 49.9 % – Everyday tasks
DeepSeek V4.1 Flash · High Cost 2.3× · Score 57.9 %
121
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Target 56.0 % – Advanced reasoning
DeepSeek V4.1 Flash 1017 build · Low Cost 2.9× · Score 61.3 %
119
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Target 62.3 % – Cutting-Edge thinking
DeepSeek V4.1 Flash 1017 build · Max Cost 4.3× · Score 62.6 %
89
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.
👑 Best overall
DeepSeek V4.1 Flash · High
Cost 2.3× · Score 57.9 % · Value index 121

Picked as the frontier couple that sits furthest below the price trend : it costs 1.2× less than the trend charges for its quality — or, read on the other axis, it scores 2.8 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.

Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — DeepSeek effort levels compared

Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.

Modelthinking offlowmediumhighxhighmaxExpected score
top effort
DeepSeek V4.1 Flash—
3.52.5–4.9off-peak 1.8×
—
2.31.5–3.6off-peak 1.1×
—
5.84.5–7.6off-peak 2.9×
63.3 %
DeepSeek V4.1 Flash 1017 build
1.00.58–1.7off-peak 0.50×
2.92.2–3.8off-peak 1.5×
—
3.42.7–4.4off-peak 1.7×
—
4.33.4–5.4off-peak 2.1×
62.6 %

"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. A model measured at a single setting (solo : no effort parameter, see the measurement sources) fills one merged cell. A thinking off column appears when a model is measured with its thinking switched off, a minimal one for the lowest level some models have below low; each is a configuration of its own, read from its own measurements. n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).

Value-index matrix — every model and effort against the price trend

The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.

Modelthinking offlowmediumhighxhighmax
DeepSeek V4.1 Flash—
7755–109off-peak 154
—
12177–190off-peak 242
—
6852–90off-peak 137
DeepSeek V4.1 Flash 1017 build
9552–176off-peak 191
11989–160off-peak 238
—
10076–131off-peak 200
—
8970–114off-peak 178

Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.

What moves everything — four factors behind the numbers

The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.

Task type

Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.

Complexity

The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).

Generation

Each release buys the same quality for less: turn on Older models to set each family's earlier models beside its current one, on the same scales.

Cache & harness

Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.

Data Sources

Every source below measured at least two (model, effort) couples on the same task; 10115 cost and 23548 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.

SourceActual config (harness · effort)TypeLinked couples (same task)
AA CritPt
artificialanalysis
AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓sweepDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA GDP-PDF
artificialanalysis
AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓sweepDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
AA GDPval v2
artificialanalysis
AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓sweepDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA HLE
artificialanalysis
AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓sweepDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA Index
artificialanalysis
AA-Index · cross-model at max + Opus 5 sweep low→max ✓sweepDeepSeek V4 Pro preview·high
AA MLCR
artificialanalysis
AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓sweepDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA Omniscience
artificialanalysis
AA model payload · Omniscience accuracy (not the penalised index) ✓sweepDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA SciCode
artificialanalysis
AA model payload · SciCode, 4 models × ladder + 3 at max ✓sweepDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA TB 4.0
artificialanalysis
AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓sweepDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
AA Terminal-Bench-Science
artificialanalysis
AA model payload · Terminal-Bench-Science 0.1 (AA harness, separate from Vals's board), score-only, 4 models × ladder + 2 at max ✓sweepDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
AA-Briefcase
artificialanalysis
AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓sweepDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA-LCR
artificialanalysis
AA model payload · long-context reasoning, input-dominated cost ✓sweepDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
ARC-AGI-1
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓sweepDeepSeek V4 Pro 0813 build·low · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·nothink
ARC-AGI-2
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓sweepDeepSeek V4 Pro 0813 build·low · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·nothink
bug-hunt-bench
github-bughuntbench
Claude Code 2.1.251 · 105 planted bugs, blind judge, 3 models × ladder ✓sweepDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low
BullshitBench v2
github-bullshitbench
petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓sweepDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·low · DeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·low
CursorBench 4.0
cursorbench
cursor.com payload · v4.0 long-horizon suite, Opus 5/5.5, Fable 5.1, Sonnet 5/5.5 × low→max, measured tokens × list incl. cache ✓sweepDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low
DeepSeek V4.1 tech report effort curve: AIME 2026
deepseek
AIME 2026, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓sweepDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high
DeepSeek V4.1 tech report effort curve: Apex 2025 Shortlist
deepseek
Apex 2025 Shortlist, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓sweepDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
DeepSeek V4.1 tech report effort curve: DeepSWE v1.1 DeepSeek Harness Minimal
deepseek
DeepSWE v1.1 DeepSeek Harness Minimal, deepseek-v4.1-flash at reasoning effort 25..100 (Fig11); Pass@1 ; harness DeepSeek-Harness-minimal; max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓sweepDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
DeepSeek V4.1 tech report effort curve: HLE-Text
deepseek
HLE-Text, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓sweepDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
DeepSeek V4.1 tech report effort curve: IMO-AnswerBench
deepseek
IMO-AnswerBench, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓sweepDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
DeepSeek V4.1 tech report effort curve: LiveCodeBench
deepseek
LiveCodeBench, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓sweepDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
DeepSeek V4.1 tech report effort curve: SimpleQA-Verified
deepseek
SimpleQA-Verified, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓sweepDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
FrontierCode main
cognition
Cognition leaderboard JSON · v1.1 main (100 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓sweepDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash·high
FrontierCode-Ext
cognition
Cognition leaderboard JSON · v1.1 Extended (150 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓sweepDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash·high
ibragim.dev · Pi
ibragim-dev
Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Pi 0.85.1, effort per row, pass@1 and $/task; board of 28 Sep ✓sweepDeepSeek V4.1 Flash·max
Vals VibeCode
vals.ai
Vals AI · Opus 5 sweep low→max (scores), $ cost at leaderboard setting; board entries — snapshot 27 Sep, re-priced ✓sweepDeepSeek V4.1 Flash 1017 build·max
vlm-exam
github-vlmexam
roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓sweepDeepSeek V4 Flash 0731 build·high
WeirdML
epoch-archive
Epoch archive · 3 models × high/max, cost per run ✓sweepDeepSeek V4 Pro 0813 build·max · DeepSeek V4 Flash 0731 build·high
deepseek-v41-techreport: deepswe
deepseek
DeepSWE v1.1 (Resolved), mini-SWE harness; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness mini-SWE; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓sweep+xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: gpqa
deepseek
GPQA Diamond (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓sweep+xmodelDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
deepseek-v41-techreport: mathapex
deepseek
MathArena Apex (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓sweep+xmodelDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
deepseek-v41-techreport: tb21
deepseek
Terminal-Bench 2.1 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓sweep+xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high
FeatherBench
ed-is-ai/featherbench
OpenRouter(Anthropic) · 28-task multi-domain suite (coding/data/realworld/security/tool-use), effort=high (Haiku=solo), mean cost/tokens per task-run, consolidated results/summary.json; only Opus 5.5 rung saturated ✓xgenDeepSeek V4 Pro preview·high · DeepSeek V4.1 Flash·high
34bench, single Thai bus-route scan transcription, 5 runs/model (dtinth/34bench)
dtinth/34bench
results.svg (table-in-SVG, parsed as text) for median accuracy (1-weighted-CER over header/forward/return/footer); meta/-.json cost field (OpenRouter USD, independently re-derived here as the median of the 5 runs' own `cost` fields) for cost; effort: meta.json has "effort": null and README says each model runs at its default reasoning effort, routed through OpenRouter; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
AA /task v3
artificialanalysis
AA-Index v3 · max ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
AA AutomationB.
artificialanalysis
AA evaluation page · 657 tasks partial-credit, 3 models at max ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
AA FrontierCode 1.1 (AA-run, weighted score)
artificialanalysis
Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
AA MMMU-Pro
artificialanalysis
Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max
AA-AnalystAgent
artificialanalysis
Artificial Analysis AA-AnalystAgent; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
aa-index411
tabbit.ai
—xmodelDeepSeek V4 Pro 0813 build·max
aa-index5
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·nothink
aa5-aa-briefcase
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
aa5-critpt-v2-hard
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
aa5-frontiercode-2-0
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
aa5-hle-diamond
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
aa5-mka
artificialanalysis
—xmodelDeepSeek V4.1 Flash 1017 build·max
aa5-mlcr-aa
artificialanalysis
—xmodelDeepSeek V4.1 Flash 1017 build·max
aa5-omniscience-non-hallucination
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
aa5-vials-aa
artificialanalysis
—xmodelDeepSeek V4.1 Flash 1017 build·max
aaenterpriseops
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max
aagpqa
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max
aaomni-index
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
aatau-banking
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max
aatb21
artificialanalysis
—xmodelDeepSeek V4 Pro 0813 build·max
AI Coding Daily LLM Coding Leaderboard (4 projects, max 70)
aicodingdaily
Laravel 20 + React-TS 20 + bug finding 10 + edge-case 20 = 70; Claude Code rows; author-scored; page 'Last update: October 17th, 2026'; read 2026-10-18 ✓xmodelDeepSeek V4.1 Flash 1017 build·max
aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high)
aibenchy
aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high
ALE-Bench
epoch-archive
Epoch archive · 3 models at matched high, measured token splits ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
ARC-AGI-3 (semi-private)
arcprize
ARC Prize ARC-AGI-3 semi-private interactive-game set, score in % (as published x100), cost = total run USD; Haiku 5.5 on the Standard harness (the Provider-Adapter harness entry, 2.82 % / $437.14, is a different configuration and is not mixed in); read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash 1017 build·max
arxiv-2610.12596-spirebench-acts
arxiv-2610.12596
—xmodelDeepSeek V4.1 Flash·max
arxiv-2610.13075-longbench-v2-direct
arxiv-2610.13075
—xmodelDeepSeek V4 Pro 0813 build·max
arxiv-2610.13075-officeqa-v1-direct
arxiv-2610.13075
—xmodelDeepSeek V4 Pro 0813 build·max
arxiv-2610.13075-officeqa-v2-direct
arxiv-2610.13075
—xmodelDeepSeek V4 Pro 0813 build·max
arxiv-2610.14397-realitybench-doc
arxiv-2610.14397
—xmodelDeepSeek V4.1 Flash·high
arxiv-2610.25078-homebench-im-f1
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-homebench-im-succ
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-homebench-vm-f1
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-homebench-vm-succ
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-bird-acf1-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-bird-safedeny-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-bird-safeex-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-bird-viol-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-spider-acf1-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-spider-safedeny-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-spider-safeex-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-rbac-spider-viol-direct
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-traceelephant-when
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-traceelephant-who
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-ww-algo-when
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-ww-algo-who
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-ww-hand-when
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-ww-hand-who
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-ww-pro-when
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.25078-ww-pro-who
arxiv-2610.25078
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
arxiv-2610.35671-cotcontrol-baseline
arxiv-2610.35671
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
arxiv-2610.35671-cotcontrol-constrained
arxiv-2610.35671
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
arxiv-2610.35671-ncri
arxiv-2610.35671
—xmodelDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·low · DeepSeek V4 Pro 0813 build·max
arxiv-2610.35671-paperbench-codedev
arxiv-2610.35671
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
arxiv-2610.35671-rebench-rust
arxiv-2610.35671
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
arxiv-2611.06135-p1-hit-sample49
arxiv-2611.06135
—xmodelDeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·nothink
arxiv-2611.06135-p1-hit-vul
arxiv-2611.06135
—xmodelDeepSeek V4.1 Flash 1017 build·high
arxiv-2611.06135-p1-mcc-all
arxiv-2611.06135
—xmodelDeepSeek V4.1 Flash 1017 build·high
arxiv-2611.06135-p1-mcc-vul
arxiv-2611.06135
—xmodelDeepSeek V4.1 Flash 1017 build·high
arxiv-2611.06135-p2-hit-vul
arxiv-2611.06135
—xmodelDeepSeek V4.1 Flash 1017 build·high
arxiv-2611.06135-p2-mcc-vul
arxiv-2611.06135
—xmodelDeepSeek V4.1 Flash 1017 build·high
AutoBCI (2609.35456) — agentic EEG architecture search, Claude Code, 6-round search
arxiv-2609.35456
Table 3, average test bAcc%/wF1/Kappa over MI/emotion/sleep held-out sets; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
bangla-llm-bench, belebele track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
bangla-llm-bench, bnexam track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
bangla-llm-bench, mgsm track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=250 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
bangla-llm-bench, mmlu track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=285 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
benchmarklist-arc-agi-1-of-105
benchmarklist
—xmodelDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
benchmarklist-arc-agi-2-of-109
benchmarklist
—xmodelDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
blog-minimax-m31f-automation
minimax
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high
blog-minimax-m31f-deepswe
minimax
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high
bsb-v1-green
github-bullshitbench-v1
—xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·low · DeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·low
cliphou.se Motion Graphics Benchmark v1.1: cost per run (Remotion videos)
cliphou.se
3 briefs x 2 runs via OpenRouter, reasoning high; cost = mean OpenRouter generation cost per run over the 6 runs (computed by us); no quality score published (human review pending); read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·high
Code4Scene private set (95 cases), overall S_model
arxiv-2609.36777
Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; private set (95 cases), overall S_model; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·high
Code4Scene public set (95 cases), overall S_model
arxiv-2609.36777
Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; public set (95 cases), overall S_model; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·high
Codex Radar intelligence-efficiency board, Codex CLI harness
codexradar
Codex Radar recurring repository-level coding eval (per-task pass over valid tasks; payload mode equal_latest_3: equal-weight latest three valid samples per task); harness codex; snapshot payload stamped 2026-10-01T23:00:27+08:00 read 2026-10-20; score=passed/valid_tasks (derived; payload gives passed, valid_tasks, iq=150*pass_rate); cost=average_price_usd per run (estimated USD-equivalent from token prices); valid_tasks differ per row (36..336); read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high
Conceptual Reasoning Index · DTBench
conceptualreasoning.ai
Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
Conceptual Reasoning Index · LMCA
conceptualreasoning.ai
Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
Convex evals (no guidelines, web research)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines, web research' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelDeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high
Convex evals (no guidelines)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
Convex evals (with guidelines)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'with guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelDeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high
Convex LLM leaderboard v92a60e0b · no guidelines
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines; read 2026-10-15 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
Convex LLM leaderboard v92a60e0b · no guidelines + web
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines + web; read 2026-10-15 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
Convex LLM leaderboard v92a60e0b · with guidelines
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, with guidelines; read 2026-10-15 ✓xmodelDeepSeek V4 Pro preview·high
ctala
github-ctala
Claude Code CLI · reasoning not configured, temp 0.7 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Flash 0731 build·high
CTI-REALM unconstrained (300-message budget): overall reward
ctirealm-2610.14180
Unconstrained CTI-REALM (Averlon), Table 2: 50 tasks, one sample each, common harness, judges Gemma-4-31B thinking / DeepSeek-v4-flash / GLM-5.3-flash x3 trials, message budget 300 (14 models have samples run at 300 incl. 10 spliced, 12 carried from lower budgets with zero capped samples); overall weighted reward; checkpoints C0-C4 same runs (rule 8); cost = cost of the 50-task run at list prices; arXiv 2610.14180 (Oct 2026); read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·high
DataLLM Lab: 9 executed Python tasks, Qwen3.8 Max family (Prime, Max, Max-0902), pass and cost per 1,000 tasks
datallmlab
9 executed Python tasks (python3 -I subprocess, 12 s timeout), temperature 0, max_tokens 4000, one scored attempt, via OpenRouter; three runs on different days (Prime 2026-10-02, Max 2026-08-20, Max-0902 2026-09-16); cost = measured tokens x list price on the priced-on date (source-computed, 'not a billing statement'); read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·high
DeepSeek V4 preview card: apex
deepseek
apex: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: apexshort
deepseek
apexshort: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: browsecomp
deepseek
browsecomp: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: codeforces
deepseek
codeforces: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: corpusqa
deepseek
corpusqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: csimpleqa
deepseek
csimpleqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: gpqa
deepseek
gpqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: hle
deepseek
hle: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: hle-tools
deepseek
hle-tools: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: hmmt26
deepseek
hmmt26: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: imo
deepseek
imo: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: lcb
deepseek
lcb: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: mcpatlas
deepseek
mcpatlas: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: mmlupro
deepseek
mmlupro: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: mrcr
deepseek
mrcr: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: simpleqav
deepseek
simpleqav: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: swemulti
deepseek
swemulti: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: swepro
deepseek
swepro: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: sweverified
deepseek
sweverified: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: tb20
deepseek
tb20: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
DeepSeek V4 preview card: toolathlon
deepseek
toolathlon: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelDeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high
deepseek-v4-flash-0731-card: deepswe
deepseek
DeepSWE (v1.1 per later cards); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓xmodelDeepSeek V4 Pro 0813 build·max
deepseek-v4-flash-0731-card: nl2repo
deepseek
NL2Repo; DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓xmodelDeepSeek V4 Pro 0813 build·max
deepseek-v4-flash-0731-card: tb21
deepseek
Terminal-Bench 2.1 (Pass@1); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓xmodelDeepSeek V4 Pro 0813 build·max
deepseek-v41-techreport: ale
deepseek
Agents' Last Exam (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold(ALE-CLI); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: autobench
deepseek
AutomationBench v1.0.6 public set (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: cybergym
deepseek
CyberGym (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: exploitgym
deepseek
ExploitGym (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: hle
deepseek
HLE (Pass@1), full set; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: hle-tools
deepseek
HLE with tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: nl2repo
deepseek
NL2Repo-Bench (Score); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: programbench
deepseek
ProgramBench (Almost@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: tb30
deepseek
Terminal-Bench 3.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
deepseek-v41-techreport: tb40
deepseek
Terminal-Bench 4.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max
Design Creativity Bench (UI generation; overall T-score, median cost per design)
arxiv-2610.11539
80 UI design tasks / 160 prompts, one generation per design, highest effort (Opus 5.5 at xhigh); Tables 2 and 9 of arXiv 2610.11539; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
Dyad app-builder · Deskhero
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Deskhero (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelDeepSeek V4 Flash 0731 build·high
Dyad app-builder · Portalis
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Portalis (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelDeepSeek V4 Flash 0731 build·high
Dyad app-builder · Relay CRM
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Relay CRM (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelDeepSeek V4 Flash 0731 build·high
econ-eval leaderboard A (20 tasks, Opus-5 judge)
github-deluair-econ-eval
deluair/econ-eval README at commit 77f9f3e (16 Oct 2026), leaderboard of 2026-09-10: 20 economics tasks (quantitative, reasoning, coding, writing) x 5 samples = 100 completions/model, numeric/exact/code-exec graders + rubric tasks judged by Claude Opus 5; score = mean over tasks of task means; cost = total for the 100 completions from token counts x list price (prices read from the OpenRouter catalog 2026-07-31 / vendor pages); per-track tables are the same completions (not extra groups); ten models ran 2026-08-01, deepseek-flash added 2026-09-10 same tasks/samples/judge; read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Flash 0731 build·high
econ-eval leaderboard B (50 tasks, GPT-6 Astra judge)
github-deluair-econ-eval
deluair/econ-eval README at commit 77f9f3e, leaderboard B: 50 tasks (the 20 above + 30 daily-work tasks added 2026-09-15), graded by GPT-6 Astra; 5 samples per task (Argon 1); runs dated 2026-09-15 / 2026-10-13 / 2026-10-15; cost only where an API price exists; different judge and task set from leaderboard A: separate group; read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·high
EMB
vals.ai
Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max
Epoch · chess puzzles
epoch.ai
Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·high
Epoch · frontiermath tier 4 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
Epoch · frontiermath tiers 1 3 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
Epoch · furniture assembly
epoch.ai
Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash·max
Epoch · gpqa diamond
epoch.ai
Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
Epoch · mystery game puzzles
epoch.ai
Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
Epoch · otis mock aime 2024 2025
epoch.ai
Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
Epoch · simpleqa verified
epoch.ai
Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
epoch-ext-surface-evolver-bench
epoch.ai
—xmodelDeepSeek V4.1 Flash·high
fcode20pass
cognition
—xmodelDeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
fcode20score
cognition
—xmodelDeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
github-telecom-oam-scenario-on41
github-telecom-oam
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
hehee9 Maze Bench Tier 1 (30 mazes)
hehee9maze-bcd609c
Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 30 mazes (6 sizes x 5, 4x4..18x18), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status completed); read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·high
Hex DataBench
hex.tech
Hex DataBench, 100 analytics tasks in the Hex workspace, rubric-judged · score in %, average cost per task (USD), tokens per task (one total); effort from the row label; Opus 5, Fable 5.1, Fable 5, Opus 4.8, Sonnet 5 at every listed effort (Opus 5.5 in its own group); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·low
hf-kimi-k3.1-pro-aa-briefcase
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-ale-all
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-ale-cli
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-apex-agents
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-automationbench-aa
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-babyvision-py
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-benchcad-notools
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-benchcad-tools
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-biomystery
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-business-operation
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-chartography-py
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-charxiv
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-charxiv-py
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-deck-arena
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-deepswe
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-design-bench
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-frontier-finance
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-gdppdf
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-gdppdf-py
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-gdpval-aa
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-gpqa-diamond
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-hallu-hard-tools
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-hallu-notools
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-handbook
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-harvey-legal
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-hle-diamond
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-hle-full
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-hle-tools
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-kcb2
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-kcb3
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-mmmupro
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-mmmupro-py
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-officeqa-pro
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-officeqa-pro-v2
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-osworld-v21
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-perceptionbench
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-posttrainbench
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-programbench
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-swemarathon
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-tb4
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-xdev-hard
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-zerobench-avg5
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-zerobench-pass5
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
hf-kimi-k3.1-pro-zerobench-py-pass5
moonshot
—xmodelDeepSeek V4.1 Flash 1017 build·max
ibmartian-full-p-other
ironbee
—xmodelDeepSeek V4.1 Flash 1017 build·high · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4 Pro 0813 build·low
IdentityBench v1: identity-governance decisions (680 test cases), Claude CLI baseline prompt
github-identitybench
rlwillen0121/identity-eval-harness README leaderboard; frozen test split, baseline prompt, provider claude_cli; no effort set; score = grounded accuracy %; cost = API-equivalent USD per 1,000 decisions; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·high
IdentityBench v3-Full: 360 multi-turn stateful IGA episodes (max 12 turns), claude_cli provider, 17 Oct 2026
github-identitybench
rlwillen0121/identity-eval-harness README fad1304 22-model v3_full table; experiments/v3-full.yaml; NEW GROUP (held identitybench-v1 is the 680-case single-prompt v1 suite); read 2026-10-18 ✓xmodelDeepSeek V4.1 Flash·high
IdentityBench v4.1 v41_lite
github-identitybench
IdentityBench v4.1 v41_lite: 60 enterprise IGA scenarios x 6 governance categories = 360 category tasks, 1,628 required actions, 12-turn cap (T0 single monolith), seed 20261019, 1 run per model, README of 2026-10-20 (commit 56591d0), published 2026-10-19; metric tasks solved of 360; cost = run cost of the whole suite (billed or estimated); read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·high
ITBench-AA (SRE)
artificialanalysis
Artificial Analysis ITBench-AA (SRE); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
lb2-browser-analysis-driven-action
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-browser-constraint-adherence
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-browser-data-aggregation
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-browser-sequential-planning
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-browser-structural-integrity
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-coding-agent-loop
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-coding-billing-limits
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-coding-hard-fixes
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-coding-llm-api-integration
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-coding-product-ux
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-dataanalysis-consecutive-events
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-dataanalysis-table-join
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-design-documents-docx
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-design-pdf
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-design-presentations-ppt
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-longhorizon-delegated-operations
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-longhorizon-finance-compliance-apps
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-longhorizon-long-run-operations
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-longhorizon-operations-commerce-apps
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-longhorizon-performance-optimisation
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-longhorizon-systems-engineering
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-reasoning-logic-with-navigation
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-reasoning-spatial
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-reasoning-theory-of-mind
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-research-legal
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-research-regulatory-compliance
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-research-science-evidence
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-research-strategy
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lb2-research-tax-finance
livebench
—xmodelDeepSeek V4.1 Flash 1017 build·max
lechmazur Adversarial LLM Debate Benchmark — Bradley-Terry rating
lechmazur-debate
lechmazur/debate, side-swapped adversarial debates, multi-model judge panel (incl. a Claude judge), Bradley-Terry rating with match counts; read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high
lechmazur Creative Writing Benchmark — comparison score
lechmazur-writing
lechmazur/writing, matched creative-brief stories judged pairwise by rotating LLM evaluators, Thurstone/BT-style comparison score centered at 0, bootstrap 95% CI; read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·high
lechmazur NYT Connections Extended Benchmark — full 940-puzzle board
lechmazur-nyt
lechmazur/nyt-connections, 940 puzzles, quadratic-v1 scoring (g/4)^2 per puzzle, objective (not model-judged); read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·high
lechmazur NYT Connections Extended Benchmark — harder 100-puzzle subset
lechmazur-nyt
lechmazur/nyt-connections, harder 100-puzzle subset, same quadratic-v1 scoring, objective; read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·high
LiveBench 2.0 · Browser Use suite
livebench
LiveBench 2.0 Browser Use suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LiveBench 2.0 · Coding suite
livebench
LiveBench 2.0 Coding suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LiveBench 2.0 · Data Analysis suite
livebench
LiveBench 2.0 Data Analysis suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LiveBench 2.0 · Design suite
livebench
LiveBench 2.0 Design suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LiveBench 2.0 · Long-Horizon suite
livebench
LiveBench 2.0 Long-Horizon suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LiveBench 2.0 · Reasoning suite
livebench
LiveBench 2.0 Reasoning suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LiveBench 2.0 · Research suite
livebench
LiveBench 2.0 Research suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LLM A-share trading simulation (mean return)
github-llm-quant-trading
CollinsLiu615/llm-quant-trading (GitHub, commits 30 Sep 2026): simulated A-share market (GARCH, T+1), 15 stocks, 180 trading days, 100,000 CNY, 2 seeds (Claude Sonnet 5: 1 seed); score = mean total return %; individual hobby benchmark; read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·low · DeepSeek V4 Flash 0731 build·high
llm2014-logic-2026-10-median
llm2014-logic
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
LLMConfBench
arxiv-2609.20666
arXiv 2609.20666 · conformer energy ranking, 27 molecules × 3 prompts via OpenRouter, Opus 5 and Sonnet 5 at high; cost summed from the repo's per-call logs ✓xmodelDeepSeek V4 Flash 0731 build·high
LMArena Agent
lmarena.ai
Agent Mode real sessions (Code/Chat/Work) · cost=median $/task (14d window), score=Net Improvement composite signal, effort read from model-variant label; live board, read 27 Sep 2026 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
LMArena Text
lmarena.ai
Text arena (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-opus-5.5-high-vertex); live board, read 30 Sep 2026 ✓xmodelDeepSeek V4.1 Flash·max
LMArena Vision
lmarena.ai
Vision arena (Overall, the page's default board: style control) · blind pairwise votes on image-input prompts, Elo rating, no cost; vote cutoff 28 Sep 2026, no 5.5 model yet; effort read from the arena's model key (-thinking, -thinking-16k/32k, -high, -max); live board, read 30 Sep 2026 ✓xmodelDeepSeek V4.1 Flash·max
LMArena WebDev
lmarena.ai
Code arena, WebDev board (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-sonnet-5-5-high); live board, read 30 Sep 2026 ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max
LMArena WebDev — HTML sub-category
lmarena.ai
lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-html, latest split, publish 2026-10-02; read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·max
LMArena WebDev — React sub-category
lmarena.ai
lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-react, latest split, publish 2026-10-02; read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·max
M-GATE grammar error detection, MCC mean over 30 languages (temperature 0)
mgate-2608.03803
M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.1; 53 models / 82 configurations March-July 2026; linguist-crafted adversarial grammar items, 30 languages, temperature 0, one run; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓xmodelDeepSeek V4 Pro preview·high
M-GATE round-trip translation, hard subset, normalised score mean over 29 targets
mgate-2608.03803
M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.2; 53 models / 82 configurations March-July 2026; 100 English sources round-tripped into 29 languages, hard subset of 50 sentences, judges GPT-5.4 Mini + Claude Haiku 4.5 + Gemini 3.1 Flash Lite at temp 0; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓xmodelDeepSeek V4 Pro preview·high
MathArena · april
matharena.ai
MathArena, arxiv--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
MathArena · august
matharena.ai
MathArena, arxiv--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max
MathArena · august
matharena.ai
MathArena, arxiv_false--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max
MathArena · june
matharena.ai
MathArena, arxiv--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
MathArena · june
matharena.ai
MathArena, arxiv_false--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
MathArena · may
matharena.ai
MathArena, arxiv--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
MathArena · may
matharena.ai
MathArena, arxiv_false--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
matharena-arxiv-false-october
matharena.ai
—xmodelDeepSeek V4.1 Flash 1017 build·max
matharena-arxiv-false-september
matharena.ai
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max
matharena-arxiv-october
matharena.ai
—xmodelDeepSeek V4.1 Flash 1017 build·max
matharena-arxiv-september
matharena.ai
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max
matharena-arxivlean-september
matharena.ai
—xmodelDeepSeek V4.1 Flash·max
matharena-leanlean-oct26
matharena.ai
—xmodelDeepSeek V4.1 Flash 1017 build·max
MazeBench ASCII with Python tools
mazebench
MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, Python tools enabled; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓xmodelDeepSeek V4 Pro 0813 build·high
Mercor · AdvancedIF
mercor.com
Mercor, AdvancedIF: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
Mercor · BrowseComp Extended
mercor.com
Mercor, BrowseComp Extended: the public benchmark extended with new expert-built Mercor tasks (100 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
Mercor · DeepResearch Bench II
mercor.com
Mercor, DeepResearch Bench II: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
Mercor · DeepSWE v1.1
mercor.com
Mercor, DeepSWE v1.1: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
Mercor · GDPval
mercor.com
Mercor, GDPval: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
Mercor · SWE Atlas - Codebase QnA
mercor.com
Mercor, SWE Atlas - Codebase QnA: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
Mercor · SWE Atlas - Codebase QnA Extended
mercor.com
Mercor, SWE Atlas - Codebase QnA Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max
Mercor · SWE-bench Multilingual
mercor.com
Mercor, SWE-bench Multilingual: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
Mercor · SWE-bench Verified Extended
mercor.com
Mercor, SWE-bench Verified Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
Mercor APEX-Accounting
mercor.com
Mercor's own board, APEX-Accounting, harness loop_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
Mercor APEX-Agents
mercor.com
Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
Mercor APEX-v1 (The AI Productivity Index)
mercor.com
Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
mercor-apexacct-mean
mercor.com
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max
mercor-apexv1-mean
mercor.com
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
Mine Odyssey: Minecraft spatial navigation (success rate, 180 tasks)
arxiv-2610.11328
180 navigation tasks (125 outdoor/55 indoor), 500 decisions or 6 h; Opus 5 at 4 efforts plus Opus 5.5 max; Table 1 of arXiv 2610.11328; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
MineBench voxel arena
minebench
minebench.ai live API · voxel builds, 15 prompts, Elo arena read 29 Sep, measured cost for 8 of 12 Claude models ✓xmodelDeepSeek V4.1 Flash·max
NonoBench
nonobench.com
NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓xmodelDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max
NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections)
matsonj/eval-connections
results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash·low · DeepSeek V4 Flash preview·high
ockbench-top200
ockbench
—xmodelDeepSeek V4 Pro preview·high
ParseBench document-parsing leaderboard (run-llama/ParseBench)
run-llama/ParseBench
leaderboard.csv at commit afb36bd8 (2026-09-29); Overall score (Tables/Charts/ContentFaithfulness/SemanticFormatting/VisualGrounding average); cost is USD-cents per page from the repo's own pricing table (not independently verified against token counts, so flagged list-price-cost); n (documents) and run count not given in the csv; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
pokebench.tv: Pokemon Red speedrun to Brock, turns/time/cost
pokebench
fresh-save Pokemon Red, screenshot+RAM state per turn, budget 1000 turns, Anthropic API list-price cost; https://pokebench.tv/runs.json, src/pokebench_runs.json; read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·high
pokelike llm-bench harness v4 (badges mean, 50 seeds)
pokelike
pokelike llm-bench harness v4 (Pokemon-style game, 50 seeds 10000-10049, 1 pass per model, set notes=12 only); metric badges_mean of 50 runs (0-8 cap); tokens per run from the pass JSON; $/run computed by the README table from published per-token prices at print time; runs 2026-08-21 and 2026-08-22; saved repo pierpierpy/pokelike.xyz.bot; read 2026-10-20 ✓xmodelDeepSeek V4 Flash 0731 build·high
reactbench
reactbench
—xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
reactbench-fix
reactbench
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·max
reactbench-write
reactbench
—xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
RealityBench acceptability (1,378 frames)
arxiv-2610.14397
zero-shot VLM acceptability F1 against the rule scorer; cost = whole set at developer list price, thinking at output rate; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·high
ReguSim: financial manipulation detection, overall F1
arxiv-2608.19974
ReguSim (HKUST, arXiv 2608.19974 v2 12 Oct 2026) Table 5: 900 targets, LLMs receive target-marked windows with neutral account identifiers; overall F1 (%) over 45 cells; supplemental campaign of 29-30 Sep 2026 through one hosted gateway (zctotoken.com); Flash 99.3% valid outputs; read 2026-10-20 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 专业能力 (professional ability)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/专业能力.md (专业能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 医疗与心理健康 (medical/mental health)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/医疗与心理健康.md (医疗与心理健康); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 总分 (total)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/total.md (alldata); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 推理与数学计算 (reasoning/math)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/推理与数学计算.md (推理与数学计算); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 教育 (education)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/教育.md (教育); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 法律与行政公务 (law/admin)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/法律与行政公务.md (法律与行政公务); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 语言与指令遵从 (language/IF)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/语言与指令遵从.md (语言与指令遵从); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 通用能力 (general ability)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/通用能力.md (通用能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
ReLE 金融 (finance)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/金融.md (金融); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
roblox-leetcode-easy
hf-roblox-llm-leaderboard
—xmodelDeepSeek V4 Pro 0813 build·high
roblox-leetcode-hard
hf-roblox-llm-leaderboard
—xmodelDeepSeek V4 Pro 0813 build·high
roblox-leetcode-medium
hf-roblox-llm-leaderboard
—xmodelDeepSeek V4 Pro 0813 build·high
roblox-qa-freeform
hf-roblox-llm-leaderboard
—xmodelDeepSeek V4 Pro 0813 build·high
roblox-qa-multiple-choice
hf-roblox-llm-leaderboard
—xmodelDeepSeek V4 Pro 0813 build·high
SenseBench (WSD) · prompt p001
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelDeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·low
SenseBench (WSD) · prompt p002
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p002 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelDeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high
SpireBench: Slay the Spire Ascension 20 Ironclad, mean final floor reached
arxiv-2610.12596
10 runs per model on the same held-out seeds, themed reskin with note-taking; Table 1 and Table 26 of arXiv 2610.12596; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
stet-board24-oct
stet.sh
—xmodelDeepSeek V4.1 Flash·max
SuperCLUE 2026年7月 幻觉控制
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
SuperCLUE 2026年7月 数学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
SuperCLUE 2026年7月 智能体任务规划
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体任务规划, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
SuperCLUE 2026年7月 智能体编程
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体编程, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
SuperCLUE 2026年7月 科学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
SuperCLUE 2026年7月 精确指令遵循
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓xmodelDeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max
SuperCLUE-Terminal (Chinese agentic terminal coding)
superclue-terminal
SuperCLUE-Terminal (CLUE team, Aug 2026; primary board/WeChat article NOT reached): Chinese-language real programming tasks of 50-110 interaction rounds each, all models through a Claude Code-based generic agent framework; numbers read from two secondary pages dated 14 Aug 2026 (we0.ai, news.aibase.com) that agree; cost = reported mean per-task cost in CNY (benchmark execution cost, not provider pricing); read 2026-10-20 ✓xmodelDeepSeek V4 Pro 0813 build·max
Surface Evolver bench
yhenon-surface-evolver
yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro preview·high
surfevolver-pass
yhenon-surface-evolver
—xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Pro preview·high
Surge AI · Chartography
surgehq.ai
Surge AI Chartography leaderboard (surgehq.ai/benchmarks/chartography); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
Surge AI · ComplexConstraints
surgehq.ai
Surge AI ComplexConstraints leaderboard (surgehq.ai/benchmarks/complex-constraints); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
Surge AI · EnterpriseBench: CoreCraft
surgehq.ai
Surge AI EnterpriseBench: CoreCraft leaderboard (surgehq.ai/benchmarks/enterprisebench-corecraft); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
Surge AI · GDP.pdf
surgehq.ai
Surge AI GDP.pdf leaderboard (surgehq.ai/benchmarks/gdp-pdf); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
Surge AI · Riemann-bench
surgehq.ai
Surge AI Riemann-bench leaderboard (surgehq.ai/benchmarks/riemann-bench); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·max
swe-rebench 07/26
swe-rebench
ReAct minimal · Opus5/Fable5/Sonnet5 all at high ✓xmodelDeepSeek V4 Pro preview·high
synthbench-althing-globalopinionqa
github-synthbench
—xmodelDeepSeek V4.1 Flash·high
synthbench-althing-gss
github-synthbench
—xmodelDeepSeek V4.1 Flash·high
synthbench-althing-opinionsqa
github-synthbench
—xmodelDeepSeek V4.1 Flash·high
synthbench-althing-subpop
github-synthbench
—xmodelDeepSeek V4.1 Flash·high
synthbench-plain-globalopinionqa
github-synthbench
—xmodelDeepSeek V4.1 Flash·low
synthbench-plain-gss
github-synthbench
—xmodelDeepSeek V4.1 Flash·low
synthbench-plain-opinionsqa
github-synthbench
—xmodelDeepSeek V4.1 Flash·low
synthbench-plain-subpop
github-synthbench
—xmodelDeepSeek V4.1 Flash·low
synthorai-fix30-notools
synthorai
—xmodelDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low
synthorai-flash1017-fix30
synthorai
—xmodelDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
synthorai-oneshot13
synthorai
—xmodelDeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max
The Endless Exam (2609.24555) — 69 open-ended maths constructions, Claude Code tools off
arxiv-2609.24555
Table 2, 69 instances, overall score = 100 x mean relative quality vs reference; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high
Tinybird SQL benchmark
llm-benchmark.tinybird.live
Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high · DeepSeek V4 Flash 0731 build·high
Toolathlon-Verified leaderboard (current, 108 tasks)
toolathlon
Toolathlon-Verified (released 30 Jun 2026), 108 tasks, 32 MCP servers / 604 tools + 7 local toolkits, Pass@1 % (+-CI) over 3 runs; rows carry the 'Evaluated by us' badge unless noted; Agent 'Default'; page fetched 2026-10-20 (newest row 2026-08-30); cells identical to held cells (vendor cards/blogs copying the board) dropped; read 2026-10-20 ✓xmodelDeepSeek V4 Pro 0813 build·max
Towards AI internal writing-Elo board, Sep 22, 2026
reddit-1wr03jc-img1
10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; top-16 leaderboard bar chart, footer 'Board of Sep 22, 2026'; values are printed bar labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓xmodelDeepSeek V4.1 Flash·max
Vals Code Migration
vals.ai
Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max
Vals CorpFin v2
vals.ai
Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelDeepSeek V4 Flash 0731 build·high
Vals CyberBench
vals.ai
Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals Finance Agent v2
vals.ai
Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max
Vals GPQA
vals.ai
Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max
Vals HLAB
vals.ai
Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals Index (29 Sep)
vals.ai
vals.ai Vals Index as recomputed on 29 Sep 2026 (older models dropped), max effort, accuracy and cost per test ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max
Vals IOI
vals.ai
Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals Legal Research
vals.ai
Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
Vals LegalBench
vals.ai
Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals LiveCodeBench
vals.ai
Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max
Vals MedCode
vals.ai
Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals MedScribe
vals.ai
Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals MMLU-Pro
vals.ai
Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max
Vals MysteryMechanism
vals.ai
Vals AI · MysteryMechanism, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4.1 Flash·high
Vals ProgramBench
vals.ai
Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals ProofBench
vals.ai
Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals Public Benefits v1.1
vals.ai
Vals AI · Public Benefits v1.1, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4.1 Flash·max
Vals SAGE
vals.ai
Vals AI · grading student math work against a rubric, all at max ✓xmodelDeepSeek V4.1 Flash·high
Vals Science Index (biology data investigation, mechanism discovery, research workflows)
vals.ai
vals.ai Science Index, equal-weighted BioMysteryBench / MysteryMechanism / Terminal-Bench-Science; cost per test; read 2026-10-18 ✓xmodelDeepSeek V4.1 Flash·high
Vals SRE-Bench
vals.ai
Vals AI · binary reverse engineering (despite the name), all at max — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high
Vals SWE-bench
vals.ai
Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max
Vals Tax Agent
vals.ai
Vals AI · Tax Agent, 10 Claude entries at their stated compute_effort — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4.1 Flash 1017 build·max
Vals TaxEval v2
vals.ai
Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max
Vals TB 4.0
vals.ai
Vals AI · Terminal-Bench 4.0, all Claude at max, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelDeepSeek V4.1 Flash 1017 build·max
vals.ai SWE-bench Multimodal (JS front-end issues with screenshots)
vals.ai
Vals-run agent; resolve rate overall, cost per test (USD); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓xmodelDeepSeek V4.1 Flash·high
valscodemig11
vals.ai
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
valsemb11
vals.ai
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
valsfab21
vals.ai
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
valsfinindex
vals.ai
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
valshlab-2026-10-21
vals.ai
—xmodelDeepSeek V4 Pro 0813 build·max
valsindex4
vals.ai
—xmodelDeepSeek V4.1 Flash 1017 build·max · DeepSeek V4 Pro 0813 build·max
valsinfprovider
vals.ai
—xmodelDeepSeek V4.1 Flash·high
valslegalindex
vals.ai
—xmodelDeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·max
valssage11
vals.ai
—xmodelDeepSeek V4.1 Flash·max
valstaxwp
vals.ai
—xmodelDeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max
vlmexam-text
github-vlmexam
—xmodelDeepSeek V4 Flash 0731 build·high
WeirdML v3 (agentic, token-axis area score)
epoch-archive
11 ML tasks x 15 configurations; score = token-axis area 0-1 with 95% CI; cost = mean API $/run; agent claude_code; read 2026-10-01 ✓xmodelDeepSeek V4.1 Flash·high
How the numbers are built — the method

Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.

In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.

The procedure

  1. Every number traced to its source. Each source is recorded as facts: the value as the source writes it, the exact text or table cell it is read from, the URL and the date of the capture; agents read every value line by line, and a measurement whose facts lack their proof stays out. Registries of models (the names each was served under, its effort settings and defaults, dated), routes (makers' APIs, resellers, routers, harnesses), benchmarks and their versions, publishers and dated prices let one build decide, by a fixed decision tree, which model, configuration and price each measurement belongs to.
  2. Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale, one version of a board; a group that mixes them is split. Every run is dated, and its model and effort are read as the maker and every route in between served them on that date — never assumed: a route the source does not name is never presumed to be the maker's API, and every route that could have served the run must give the same configuration. At the least doubt on the model, the effort or what was run, the measurement stays in the data file and out of the fit.
  3. Costs at today's prices. Every cost is put on the model's current price from the prices in force on its run date: the source's own figure at that price, the price change applied to the run's own token counts, or, when the token mix is unknown, a price band whose width is added to the cost's precision. A cost that cannot be brought to the current price stays out.
  4. Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The scale is read from the metric's label, never from the benchmark's identity; whether lower is better is read from the benchmark's registry, and such a score is turned in its own unit first (an attack success rate of 30 % reads 70 %); a score whose direction is not established stays out. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
  5. One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result. The sizes of these effects are estimated for each model maker on its own, not tied to the other makers'; a maker's own charts about its own models carry an estimated home effect, also each maker's own, left out of every value. The fit takes the couples of eight makers measured in the same groups (Anthropic, OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi), on one scale; this page shows the DeepSeek couples. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
  6. Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source; the one weight set by rule is that of a single run: when a source gives k quality measurements of one run (several metrics, sub-scores or slices of the same executions), each weighs 1/k, so the run counts as one measurement.
  7. Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, a board copying another's numbers, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. Versions of one board that re-grade the same runs (recognised by identical costs) count those runs once. Before the fit, two groups that still share identical costs on three couples or more stop it until the copy is resolved or the coincidence proven from the sources. An index keeps, for a couple whose index components are already in the data, only what its missing components add, or is left out.
  8. Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
  9. From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.

Key values

  • Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
  • Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
  • Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit one figure on this page can move by one unit in that digit (the cost of DeepSeek V4 Pro preview at high), far inside their intervals.
  • Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.

Techniques and software

  • Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes, six chains whose draws continue in the same run until the criteria are met (CmdStan only to resume a stopped fit from its stored draws, or to restart a chain left in a far region from a good chain's last draw); built and checked stage by stage following the Bayesian workflow.
  • Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
  • Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
  • Scales: the empirical logit of bounded scores, with the binomial shape of its noise.

Checks (on the fit of 29 September 2026, before the other makers' couples joined the data). Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.

Alexandre Gensse · github.com/alexandregensse-blip

All figures are indicative and for informational purposes only — derived from public third-party measurements, not an official benchmark, and not affiliated with or endorsed by any model maker. Verify before relying on any value.

Data and text under CC BY 4.0 · code under MIT · download the data (CSV)