Google models compared cost vs quality at every effort level

Data analysis · aggregated from 181 sources · 929 benchmarks · 2814 measurements
GitHub
Alexandre Gensse

What each recent Google model actually costs, at every effort level, and the quality it delivers — fused from public benchmark measurements for every (model, effort) pair, on the same scales as every other maker's on this site.

The best value today is Gemini 4 Hydrogen at low effort — 3.2× cheaper than the price trend for its quality (expected score 58.3 % on the benchmark panel).

Quality is the expected score across the whole benchmark panel, not one benchmark: the type of task can move it, see what moves everything.

Measurements
181 sources · 929 benchmarks · 2814 measurements
Updated
19 Nov 2026
👑 Best overall
Gemini 4 Hydrogen · Low Cost 4.8× · Score 58.3 %
324
Grunt work
Gemini 3.1 Flash-Lite · Minimal Cost 1.0× · Score 43.4 %
229
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Everyday tasks
Gemini 4 Hydrogen · Low Cost 4.8× · Score 58.3 %
324
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Advanced reasoning
Gemini 4 Hydrogen · Medium Cost 23× · Score 67.9 %
220
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Cutting-Edge thinking
Gemini 4 Carbon · High Cost 162× · Score 75.3 %
93
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.

Quality vs. cost of every Google model, at each effort level

One point per couple: a model at one configuration (thinking off, minimal, low → max, thinking on); no line joins a model's couples. Only the couples of the Pareto frontier are labelled (model · effort), where there is room; hover or tap any point for its model, effort, cost and score. Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward.

Best Google model for the price — the Pareto frontier

The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated; the frontier couples are labelled (model · effort) where there is room — hover or tap any point for its model, effort, cost and score. No line joins a model's couples. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.

Value — distance to the price trend

The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 14 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.89, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.

Model · effortcostexpected scorevalue index
trend = 100
Gemini 4 Hydrogen · Low4.8×58.3 %324
Gemini 3.1 Flash-Lite · Medium1.9×48.1 %232
Gemini 3.1 Flash-Lite · Minimal1.0×43.4 %229
Gemini 4 Hydrogen · Medium23×67.9 %220
Gemini 3.5 Flash-Lite · Minimal2.8×48.2 %158
Gemini 4 Hydrogen · High59×71.4 %138
Gemini 4 Argon · Medium39×promo 19×68.3 %136
Gemini 4 Hydrogen · xHigh81×72.8 %122
Gemini 4 Carbon · Medium126×promo 63×74.8 %108
Gemini 4 Carbon · High162×promo 81×75.3 %93

Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.

Best value by task tier — what to actually run

Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.

Target 44.9 % – Grunt work
Gemini 3.1 Flash-Lite · Minimal Cost 1.0× · Score 43.4 %
229
Classification, tagging, extraction, routing, log/PR triage — run at scale, where throughput and unit cost dominate.
Target 54.5 % – Everyday tasks
Gemini 4 Hydrogen · Low Cost 4.8× · Score 58.3 %
324
Routine coding, refactors, unit tests, summaries, first-draft agent steps — solid work that doesn't need the frontier.
Target 65.0 % – Advanced reasoning
Gemini 4 Hydrogen · Medium Cost 23× · Score 67.9 %
220
Production code review, architecture, hard debugging, customer-facing reasoning — you need essentially flagship quality.
Target 74.1 % – Cutting-Edge thinking
Gemini 4 Carbon · High Cost 162× · Score 75.3 %
93
Research-grade reasoning, novel or ambiguous problems, the hardest agentic runs — a few extra points of capability are worth a premium.
👑 Best overall
Gemini 4 Hydrogen · Low
Cost 4.8× · Score 58.3 % · Value index 324

Picked as the frontier couple that sits furthest below the price trend : it costs 3.2× less than the trend charges for its quality — or, read on the other axis, it scores 9.5 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.

Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — Google effort levels compared

Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.

ModelminimallowmediumhighxhighmaxExpected score
top effort
Gemini 4 Carbon—
144108–192promo 72×
12695–167promo 63×
162125–209promo 81×
——75.3 %
Gemini 4 Hydrogen—
4.83.1–7.5
2318–29
5948–73
8161–108
—72.8 %
Gemini 4 Argon—
3026–35promo 15×
3933–46promo 19×
10897–119promo 54×
——70.9 %
Gemini 3.5 Flash-Lite
2.82.1–3.8
2.31.9–2.7
5.64.6–6.8
119.1–13
——51.3 %
Gemini 3.1 Flash-Lite
1.00.74–1.4
107.1–15
1.91.4–2.5
6.75.1–8.9
——45.5 %

"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. A model measured at a single setting (solo : no effort parameter, see the measurement sources) fills one merged cell. A thinking off column appears when a model is measured with its thinking switched off, a minimal one for the lowest level some models have below low; each is a configuration of its own, read from its own measurements. n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).

Value-index matrix — every model and effort against the price trend

The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.

Modelminimallowmediumhighxhighmax
Gemini 4 Carbon—
5034–73promo 99
10876–154promo 216
9368–127promo 185
——
Gemini 4 Hydrogen—
324168–626
220150–324
13895–199
12280–187
—
Gemini 4 Argon—
158124–201promo 315
136108–171promo 272
7060–82promo 140
——
Gemini 3.5 Flash-Lite
158104–238
11781–168
7550–113
6248–80
——
Gemini 3.1 Flash-Lite
229137–383
2614–47
232143–375
4629–73
——

Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.

What moves everything — four factors behind the numbers

The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.

Task type

Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.

Complexity

The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).

Generation

Each release buys the same quality for less: turn on Older models to set each family's earlier models beside its current one, on the same scales.

Cache & harness

Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.

Data Sources

Every source below measured at least two (model, effort) couples on the same task; 10115 cost and 23548 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.

SourceActual config (harness · effort)TypeLinked couples (same task)
AA CritPt
artificialanalysis
AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓sweepGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
AA GDP-PDF
artificialanalysis
AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓sweepGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
AA GDPval v2
artificialanalysis
AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓sweepGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 4 Argon·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.1 Flash-Lite·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high
AA Harvey LAB
artificialanalysis
AA model payload · Harvey's Legal Agent Bench (AA's own run, separate from Vals's HLAB), score-only, 8 models × ladder + 4 at max ✓sweepGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high
AA HLE
artificialanalysis
AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓sweepGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
AA Index
artificialanalysis
AA-Index · cross-model at max + Opus 5 sweep low→max ✓sweepGemini 3.5 Flash·high
AA Index v4
artificialanalysis
AA model pages · Fable 5.1 sweep low→max + Fable 5/Opus 5 at max, per-suite $ ✓sweepGemini 3.5 Flash·high · Gemini 3.7 Flash·high
AA MLCR
artificialanalysis
AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓sweepGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash·high
AA Omniscience
artificialanalysis
AA model payload · Omniscience accuracy (not the penalised index) ✓sweepGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3 Pro·high · Gemini 3.1 Flash-Lite·high
AA SciCode
artificialanalysis
AA model payload · SciCode, 4 models × ladder + 3 at max ✓sweepGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high
AA TB 4.0
artificialanalysis
AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓sweepGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
AA Terminal-Bench-Science
artificialanalysis
AA model payload · Terminal-Bench-Science 0.1 (AA harness, separate from Vals's board), score-only, 4 models × ladder + 2 at max ✓sweepGemini 3.8 Flash·medium · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
AA-Briefcase
artificialanalysis
AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓sweepGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.1 Flash-Lite·high
AA-LCR
artificialanalysis
AA model payload · long-context reasoning, input-dominated cost ✓sweepGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
Agents on Rails · features
rubyonrails
rubyonrails.org/ai + rails/ai-evals runs · 20 tickets ×3, miniswen; Opus 5 & Fable 5.1 high→max, Opus 5.5 default ✓sweepGemini 3.8 Flash·medium · Gemini 3.8 Flash·high
ARC-AGI-1
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓sweepGemini 3 Flash·minimal · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.6 Flash·minimal · Gemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·low · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
ARC-AGI-2
arcprize
ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓sweepGemini 3 Flash·minimal · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.6 Flash·minimal · Gemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·low · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
BBQ effort (interface-gap)
arxiv-2609.08861
direct API · budget_tokens low/med/high, Sonnet4.6 & Haiku4.5 ✓sweepGemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high
BullshitBench v2
github-bullshitbench
petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓sweepGemini 3.5 Flash-Lite·minimal · Gemini 4 Argon·low · Gemini 4 Argon·high · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Pro·low · Gemini 3 Pro·high · Gemini 3.7 Flash·low · Gemini 3.6 Flash·minimal · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·minimal · Gemini 3 Flash·high
CursorBench 3.2
cursorbench
Cursor 3.2 · matched effort, $ cost ✓ (vendor benchmark)sweepGemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high
CursorBench 4.0
cursorbench
cursor.com payload · v4.0 long-horizon suite, Opus 5/5.5, Fable 5.1, Sonnet 5/5.5 × low→max, measured tokens × list incl. cache ✓sweepGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low · Gemini 4 Hydrogen·xhigh · Gemini 4 Hydrogen·high · Gemini 4 Hydrogen·medium · Gemini 4 Hydrogen·low
DeepResearch
futuresearch
Deep Research Bench · low/high ✓sweepGemini 3 Flash·low · Gemini 3 Flash·high
FrontierCode main
cognition
Cognition leaderboard JSON · v1.1 main (100 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓sweepGemini 3.6 Flash·low · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
FrontierCode-Ext
cognition
Cognition leaderboard JSON · v1.1 Extended (150 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓sweepGemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
HellaSwag effort (interface-gap)
arxiv-2609.08861
direct API · budget_tokens low/med/high, Sonnet4.6 & Haiku4.5 ✓sweepGemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high
MCPMark
eval-sys/mcpmark
eval-sys/mcpmark legacy board · 5 MCP servers (FS/GitHub/Notion/Playwright/Postgres), Sonnet 4 low/default/high + Opus 4.1/Sonnet 4.5/Opus 4.5, Cost-per-run; low>high cost inversion kept as measured ✓sweepGemini 3 Pro·high
nnakapa #36 Copilot
zenn-nnakapa
GitHub Copilot CLI 1.0.80 · same 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, billed cost ✓sweepGemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high
ObviousBench 0.2
benchmarklist
inspect-ai · 144 items ×3 epochs, 3 models × full ladder, measured tokens ✓sweepGemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low
PetriBench
arxiv-2609.19883
arXiv 2609.19883 · 4,800 Petri-net questions, raw API adaptive thinking + output_config.effort, Opus 5 M/H/XH + Sonnet 5 M/H, measured tokens × list ✓sweepGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium · Gemini 3.1 Pro·medium
Playcode MacBook SVG
playcode
Playcode · one-shot MacBook SVG, high/xhigh/max per model, provider-billed cost (quality ordinal, not scored) ✓sweepGemini 3.6 Flash·medium · Gemini 3.8 Flash·medium · Gemini 4 Argon·medium · Gemini 4 Argon·high · Gemini 4 Argon·low
QuoteBench
arxiv-2608.13547
arXiv 2608.13547 · 56 frozen tasks, provider-reported output tokens, 3 models × ladder ✓sweepGemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high
Ramen Bench
github-ramenbench
not-stbenjam/ramen-bench · one-shot creative-coding HTML, Sonnet 5.5, Opus 5.5, Opus 5, Fable 5.1 × low→max, Claude Code run.json, cost only ✓sweepGemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 4 Argon·low · Gemini 4 Argon·medium
renkon40 · 3-minute talk slides
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, 3-minute talk slides; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepGemini 3.8 Flash·high
renkon40 · Note article from an experiment log
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, Note article from an experiment log; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepGemini 3.8 Flash·high
renkon40 · short video
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, short video (60 s from a 25-min YouTube talk); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓sweepGemini 3.8 Flash·high
Sonar Java
sonarsource
Sonar leaderboard JSON · 4,444 Java tasks, single-shot, measured tokens × list price; Opus 5.5 medium/high ✓sweepGemini 3 Flash·high · Gemini 3 Pro·high · Gemini 3.1 Flash-Lite·high · Gemini 3.1 Pro·high · Gemini 4 Argon·high
Vals VibeCode
vals.ai
Vals AI · Opus 5 sweep low→max (scores), $ cost at leaderboard setting; board entries — snapshot 27 Sep, re-priced ✓sweepGemini 4 Carbon·high
Vals Web Search (finance)
vals.ai
vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓sweepGemini 3.5 Flash·high
Vals Web Search (legal)
vals.ai
vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓sweepGemini 3.5 Flash·high
vibe-openscad
github-vibeopenscad
output_config.effort explicit · 3 models × ladder; quality saturates → cost only ✓sweepGemini 3.1 Flash-Lite·minimal · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·minimal
vlm-exam
github-vlmexam
roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓sweepGemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·high · Gemini 3 Flash·low · Gemini 3 Flash·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high · Gemini 4 Argon·low · Gemini 4 Argon·high
WeirdML
epoch-archive
Epoch archive · 3 models × high/max, cost per run ✓sweepGemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash-Lite·high
Zapier AutomationB.
benchmarklist
AutomationBench 1.0.6 · 657 held-out tasks, strict pass/fail, measured tokens × list ✓sweepGemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·high · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.6 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Hydrogen·high · Gemini 4 Hydrogen·medium · Gemini 4 Carbon·low
Deictic ambiguity ablation experiment balanced accuracy
arxiv-2609.12162
Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); ablation experiment; read 2026-10-19 ✓xgenGemini 3 Pro·high · Gemini 3 Flash·medium · Gemini 3 Flash·low · Gemini 3 Flash·minimal · Gemini 3 Flash·high
Deictic ambiguity primary experiment balanced accuracy
arxiv-2609.12162
Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); primary experiment; read 2026-10-19 ✓xgenGemini 3 Pro·high · Gemini 3 Flash·medium · Gemini 3 Flash·high · Gemini 3 Flash·low · Gemini 3 Flash·minimal
FeatherBench
ed-is-ai/featherbench
OpenRouter(Anthropic) · 28-task multi-domain suite (coding/data/realworld/security/tool-use), effort=high (Haiku=solo), mean cost/tokens per task-run, consolidated results/summary.json; only Opus 5.5 rung saturated ✓xgenGemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Google model cards: DeepSWE v1.1
google
DeepSWE v1.1 (Datacurve), mini-swe-agent harness; Google model-card table(s) 21 Jul 2026 (google-gemini-3.6-flash-card); 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.6 Flash·high · Gemini 3.5 Flash·medium
Google model cards: GPQA Diamond (no tools)
google
GPQA Diamond no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Google model cards: Humanity's Last Exam (full set, no tools)
google
HLE full set text+MM, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Google model cards: Humanity's Last Exam (search + code)
google
HLE full set text+MM, search (blocklist) + code execution; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Google model cards: LiveCodeBench Pro (Elo)
google
LiveCodeBench Pro Elo; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Google model cards: MMMLU
google
MMMLU multilingual MMLU; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Google model cards: MMMU-Pro (avg Standard + Vision, no tools)
google
MMMU-Pro averaged over Standard (10 options) and Vision, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Google model cards: MRCR v2 8-needle 128k (average)
google
MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.5 Flash-Lite·high
Google model cards: MRCR v2 8-needle 128k (average)
google
MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Google model cards: MRCR v2 8-needle 1M (pointwise)
google
MRCR v2 8-needle, 1M pointwise; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high
Google model cards: SWE-Bench Pro (Public)
google
SWE-Bench Pro public set; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high · Gemini 3 Pro·high
Google model cards: SWE-bench Verified (single attempt)
google
SWE-bench Verified, single attempt; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xgenGemini 3.1 Pro·high
Official tau2-bench leaderboard, banking_knowledge (AllTools), Claude submissions by Sierra
github-sierra-tau2-bench
banking_knowledge domain, 4 trials, user simulator gpt-5.2 (low), reasoning_effort max for all; pass^1 %; read 2026-10-18 ✓xgenGemini 3.1 Pro·high
Writing-Elo effort/cost chart, Reddit 1wrkwp0 (r/LLMDevs)
reddit-1wrkwp0
same 'Writing Elo' metric family, 10 real YouTube scripts per OP, scored blind by three judges and a rubric; cost per script, log-x scatter with effort-labelled points; no board date printed on the chart; a different Reddit post/source snapshot than 1wr03jc, kept as its own board version; read 2026-10-05 ✓xgenGemini 3.8 Flash·high
34bench, single Thai bus-route scan transcription, 5 runs/model (dtinth/34bench)
dtinth/34bench
results.svg (table-in-SVG, parsed as text) for median accuracy (1-weighted-CER over header/forward/return/footer); meta/-.json cost field (OpenRouter USD, independently re-derived here as the median of the 5 runs' own `cost` fields) for cost; effort: meta.json has "effort": null and README says each model runs at its default reasoning effort, routed through OpenRouter; read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium · Gemini 3.5 Flash-Lite·minimal
4DCodeBench: inverse-graphics-of-dynamic-scenes leaderboard (200 scenes), Claude Code agent
4dcodebench
arxiv.org/html/2610.03715v1 Figure 4 (src/main_leaderboard.png); Overall = mean of 5 family scores [0,1]; values printed on heatmap cells, read directly (not pixel-digitized); n=1 run/model, no repeats stated; read-from-image; read 2026-10-05 ✓xmodelGemini 3.8 Flash·high
AA /task v4
artificialanalysis
AA launch article · per-task $, Fable 5.1 xhigh/max vs Fable 5/Opus 5 max ✓xmodelGemini 3.5 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium
AA AutomationB.
artificialanalysis
AA evaluation page · 657 tasks partial-credit, 3 models at max ✓xmodelGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
AA FrontierCode 1.1 (AA-run, weighted score)
artificialanalysis
Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 4 Argon·high
AA MMMU-Pro
artificialanalysis
Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
AA-AnalystAgent
artificialanalysis
Artificial Analysis AA-AnalystAgent; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelGemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high
aa-index411
tabbit.ai
—xmodelGemini 3.7 Flash·high
aa-index5
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Hydrogen·low · Gemini 4 Hydrogen·medium · Gemini 4 Hydrogen·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
aa5-aa-briefcase
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
aa5-code2video-aa
artificialanalysis
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
aa5-critpt-v2-hard
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
aa5-frontiercode-2-0
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
aa5-harvey-lab-aa
artificialanalysis
—xmodelGemini 4 Argon·high
aa5-healthbench
artificialanalysis
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
aa5-healthbench-professional
artificialanalysis
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
aa5-hle-diamond
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
aa5-mka
artificialanalysis
—xmodelGemini 4 Carbon·high · Gemini 4 Argon·high
aa5-mlcr-aa
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.8 Flash·low
aa5-omniscience-non-hallucination
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.8 Flash·low
aa5-vials-aa
artificialanalysis
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
aaapex
artificialanalysis
—xmodelGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high
aaenterpriseops
artificialanalysis
—xmodelGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high
aagpqa
artificialanalysis
—xmodelGemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
aaifbench
artificialanalysis
—xmodelGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
aaime25
artificialanalysis
—xmodelGemini 3 Pro·high
aalcb
artificialanalysis
—xmodelGemini 3 Pro·high
aaomni-index
artificialanalysis
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3 Pro·high · Gemini 3.1 Flash-Lite·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
aatau-banking
artificialanalysis
—xmodelGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium
aatau2
artificialanalysis
—xmodelGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
aatb21
artificialanalysis
—xmodelGemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high
aatbhard
artificialanalysis
—xmodelGemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high
AI Coding Daily LLM Coding Leaderboard (4 projects, max 70)
aicodingdaily
Laravel 20 + React-TS 20 + bug finding 10 + edge-case 20 = 70; Claude Code rows; author-scored; page 'Last update: October 17th, 2026'; read 2026-10-18 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high)
aibenchy
aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓xmodelGemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·medium · Gemini 3 Flash·medium · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.5 Flash-Lite·high · Gemini 4 Argon·medium · Gemini 3.6 Flash·low · Gemini 3.1 Pro·medium · Gemini 3.5 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.5 Flash·medium · Gemini 3.5 Flash-Lite·medium · Gemini 3.8 Flash·low · Gemini 3 Flash·low · Gemini 3.1 Flash-Lite·medium · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·low · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·minimal
ALE-Bench
epoch-archive
Epoch archive · 3 models at matched high, measured token splits ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high
android-bench-2
google
—xmodelGemini 4 Argon·high
APEX-Agents public leaderboard rows quoted by StagedWorkspace (Pass@1)
stagedws-2608.18050
APEX-Agents public leaderboard rows quoted as published by arXiv 2608.18050 Table 2b (Pass@1 with 8 independent attempts per task), snapshot unknown (older than the held Mercor board: cells differ); same-benchmark-as:mercor-apexagents; metric Pass@1 %; read 2026-10-20 ✓xmodelGemini 3 Flash·high
APEX-Harm mean task score under attack (all 6 categories)
apexharm-2610.13623
APEX-Harm 180 attack task instances per model (15 underlying APEX-Agents tasks x 12 attack variants), APEX-Agents harness with MCP tools, 150 steps, no timer, judge/rules per paper; metric mean task score 0-1 (utility under attack); ASR/ER safety rates not extracted; arXiv 2610.13623 Table 4 (Oct 2026); cost = Table 1 total for Gemini 3.8 Flash only; read 2026-10-20 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high
apexharm-asr
apexharm-2610.13623
—xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high
ARC-AGI-3 (semi-private)
arcprize
ARC Prize ARC-AGI-3 semi-private interactive-game set, score in % (as published x100), cost = total run USD; Haiku 5.5 on the Standard harness (the Provider-Adapter harness entry, 2.82 % / $437.14, is a different configuration and is not mixed in); read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·low · Gemini 4 Argon·medium · Gemini 4 Argon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
arcagi3-provider-adapter
arcprize
—xmodelGemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low
Argo-Bench: enterprise data-agent tasks (210 tasks), Inspect AI 0.3.263 harness
argo-bench
arxiv.org/html/2610.02122v1 Table 2 (best-effort snapshot, extra-high where offered) + Table 10 (effort ladder, mean score /100, tasks completed at every level); n=1 rollout/task; score = mean task score out of 100 with a model's forecast tasks floored at 0 as a block; cost = API spend per task, excluding BigQuery (Table 2 only, not itemized by effort in Table 10); read 2026-10-05 ✓xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high
ArtifactArena model leaderboard, Design Lab Harness (DLH) Elo
arxiv-2610.06511
Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high
ArtifactArena model leaderboard, Sampling Harness (SH) Elo
arxiv-2610.06511
Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high
ArtifactArena model leaderboard, Verifier-Grounded Refinement Harness (VGH) Elo
arxiv-2610.06511
Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high
Artificial Analysis Coding Agent Index v1.4 (OpenAI GPT-6 Astra post chart, non-code)
openai
Artificial Analysis Coding Agent Index v1.4; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓xmodelGemini 3.8 Flash·high
arxiv-2604.26954-kappa
arxiv-2604.26954
—xmodelGemini 3 Flash·minimal · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high
arxiv-2604.26954-kappa-temp0-j1
arxiv-2604.26954
—xmodelGemini 3 Flash·minimal
arxiv-2604.26954-kappa-temp1-j3
arxiv-2604.26954
—xmodelGemini 3 Flash·minimal
arxiv-2604.26954-kappa-temp1-j5
arxiv-2604.26954
—xmodelGemini 3 Flash·minimal
arxiv-2604.26954-kappa-temp1-j7
arxiv-2604.26954
—xmodelGemini 3 Flash·minimal
arxiv-2609.33055-weighted-sd
arxiv-2609.33055
—xmodelGemini 3.8 Flash·medium
arxiv-2610.12596-spirebench-acts
arxiv-2610.12596
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.13075-longbench-v2-direct
arxiv-2610.13075
—xmodelGemini 3.8 Flash·high
arxiv-2610.13075-officeqa-v1-direct
arxiv-2610.13075
—xmodelGemini 3.8 Flash·high
arxiv-2610.13075-officeqa-v2-direct
arxiv-2610.13075
—xmodelGemini 3.8 Flash·high
arxiv-2610.14397-realitybench-doc
arxiv-2610.14397
—xmodelGemini 3.8 Flash·medium
arxiv-2610.17016-credit-kl-nats
arxiv-2610.17016
—xmodelGemini 3.8 Flash·medium
arxiv-2610.17016-peer-kl-nats
arxiv-2610.17016
—xmodelGemini 3.8 Flash·medium
arxiv-2610.26538-bymeal-breakfast
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-bymeal-dinner
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-bymeal-lunch
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-effort-high
arxiv-2610.26538
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
arxiv-2610.26538-effort-low
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-grams-eaten-relative-error
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t2-before-only
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t2-before-x-c
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t2-both-photos
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t2-both-valid
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t2-subtraction
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t3-1
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t3-3
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t3-3a
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t3-3b
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t5-within-800kj-before-only
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.26538-t5-within-800kj-both-photos
arxiv-2610.26538
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low
arxiv-2610.27043-t2-fb-de
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-fb-en
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-fb-es
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-fb-hi
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-ar
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-de
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-en
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-es
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-hi
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-it
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-ja
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27043-t2-mintaka-pt
arxiv-2610.27043
—xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash·high
arxiv-2610.27745-first-success
arxiv-2610.27745
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.27745-h1
arxiv-2610.27745
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.27745-h2
arxiv-2610.27745
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.27745-h4a
arxiv-2610.27745
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.27745-spread-prior
arxiv-2610.27745
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.27745-spread-reported
arxiv-2610.27745
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.27745-spread-rho
arxiv-2610.27745
—xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
arxiv-2610.30051-libero10-success
arxiv-2610.30051
—xmodelGemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low
arxiv-2610.30051-libero10-task0-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task1-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task2-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task3-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task4-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task5-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task6-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task7-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task8-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30051-libero10-task9-solved
arxiv-2610.30051
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
arxiv-2610.30116-author-difficulty
arxiv-2610.30116
—xmodelGemini 3.8 Flash·high
arxiv-2610.30116-author-hintlift
arxiv-2610.30116
—xmodelGemini 3.8 Flash·high
arxiv-2610.30116-solve
arxiv-2610.30116
—xmodelGemini 3.8 Flash·high
arxiv-2610.31986-t4-i1
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-i2
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-i3
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-ins
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-overall
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-p1
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-p2
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-p3
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-p4
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-perc
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-r1
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-r2
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-r3
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.31986-t4-reas
arxiv-2610.31986
—xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal
arxiv-2610.35671-paperbench-codedev
arxiv-2610.35671
—xmodelGemini 3.8 Flash·high
arxiv-2610.37747-found-likert
arxiv-2610.37747
—xmodelGemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high
arxiv-2610.38096-s14-robust-beams-rejected-bt
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s14-robust-beams-rejected-bu-retest
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s14-robust-beams-rejected-c4-bu
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-c4-u-accepted
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-c4-u-flagged
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-c4-u-noans
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-c4-u-rejected
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-c4-u-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-c4-u-vnoans
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-c4-u-wrong
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-t-accepted
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-t-flagged
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-t-rejected
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-t-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-t-wrong
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-u-accepted
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-u-flagged
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-u-noans
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-u-vnoans
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-main-u-wrong
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-pilot-u-accepted
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-pilot-u-rejected
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t02-pilot-u-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-t-accepted
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-t-flagged
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-t-rejected
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-t-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-u-accepted
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-u-flagged
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-u-noans
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-u-vnoans
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-main-u-wrong
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-pilot-u-rejected
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s2-t11-pilot-u-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s33-t02-feasibility-only-valid-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s33-t02-pa-limit-valid-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s33-t02-pa-note-rejected
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s33-t02-pa-note-valid-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s33-t02-s1-limit-valid-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s33-t02-s1-note-rejected
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s33-t02-s1-note-valid-solved
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s7-t02-main-max-error
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s7-t02-main-median-error
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s7-t02-pilot-max-error
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s7-t02-pilot-median-error
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s7-t11-main-max-error
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s7-t11-main-median-error
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-s7-t11-pilot-max-error
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-s7-t11-pilot-median-error
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-t1-beam
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-t1-disk
arxiv-2610.38096
—xmodelGemini 3.8 Flash·high
arxiv-2610.38096-t1-noans
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-t1-robust
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38096-t1-vbeam
arxiv-2610.38096
—xmodelGemini 4 Argon·high
arxiv-2610.38096-t3-t02large
arxiv-2610.38096
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38924-a-t2-pass1
arxiv-2610.38924
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38924-a-t2-pass3
arxiv-2610.38924
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38924-a-t2-passhat3
arxiv-2610.38924
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38924-a-t3-firstread
arxiv-2610.38924
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38924-a-t3-images
arxiv-2610.38924
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38924-a-t3-ocr
arxiv-2610.38924
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.38924-a-t3-upload
arxiv-2610.38924
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.41025-argon-autopsy-fixrate
arxiv-2610.41025
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.41025-argon-autopsy-fullyfixed
arxiv-2610.41025
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.41025-argon-autopsy-repair
arxiv-2610.41025
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.41025-argon-autopsy-valid
arxiv-2610.41025
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
arxiv-2610.41025-sup-effect-failing
arxiv-2610.41025
—xmodelGemini 3.8 Flash·high
arxiv-2610.41025-sup-effect-succeeding
arxiv-2610.41025
—xmodelGemini 3.8 Flash·high
arxiv-2610.41025-t4-net-all
arxiv-2610.41025
—xmodelGemini 3.8 Flash·high
arxiv-2610.41025-t4-net-nonqwen
arxiv-2610.41025
—xmodelGemini 3.8 Flash·high
arxiv-2610.41076-baba-difficulty-hard
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-baba-difficulty-light
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-baba-difficulty-standard
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-baba-issue-completion
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-baba-pass1-strict
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-baba-pass3
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t13-anyseed
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t14-hard
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t14-light
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t14-std
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t15-seed1-outcome
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t15-seed1-strict
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t15-seed2-outcome
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t15-seed2-strict
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t15-seed3-outcome
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2610.41076-t15-seed3-strict
arxiv-2610.41076
—xmodelGemini 3.7 Flash·high · Gemini 3.1 Pro·high
arxiv-2611.05147-gm-league-arena-baseball-cash
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-baseball-elo
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-football-cash
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-football-elo
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-hockey-cash
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-hockey-elo
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-cash
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-elo
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-gi
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-losses
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-wins
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-wins-2022
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-wins-2023
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-wins-2024
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-wins-2025
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.05147-gm-league-arena-nba-wins-2026
arxiv-2611.05147
—xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
arxiv-2611.07461-37-flash-t13-baseline-resolution
arxiv-2611.07461
—xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high
arxiv-2611.07461-37-flash-t13-delta
arxiv-2611.07461
—xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high
arxiv-2611.07461-t13-hermes-resolution
arxiv-2611.07461
—xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high
arxiv-2611.07667-doubleauction-coarse
arxiv-2611.07667
—xmodelGemini 3.1 Pro·low
arxiv-2611.07667-doubleauction-full
arxiv-2611.07667
—xmodelGemini 3.1 Pro·low
AutoBCI (2609.35456) — agentic EEG architecture search, Claude Code, 6-round search
arxiv-2609.35456
Table 3, average test bAcc%/wF1/Kappa over MI/emotion/sleep held-out sets; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
AutomationB.
arxiv-2604.18934
harness+effort NOT stated ✓xmodelGemini 3.1 Pro·high
AutomationBench (OpenAI GPT-5.6 post chart)
openai
AutomationBench pass rate, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓xmodelGemini 3.5 Flash·high
bangla-llm-bench, belebele track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
bangla-llm-bench, bnexam track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
bangla-llm-bench, mgsm track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=250 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
bangla-llm-bench, mmlu track (paired EN/BN; deluair/bangla-llm-bench)
deluair/bangla-llm-bench
scores.csv at commit 0cc4f72 (2026-09-28); n=285 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
Bio-imaging (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelGemini 3.1 Pro·high
BioMystery hard-dist (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelGemini 3.1 Pro·high
BioMystery hard-sub (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelGemini 3.1 Pro·high
BixBench3 mean artifact score (20 comp-bio tasks)
bixbench3-2608.25286
BixBench3 20 tasks / 138 artifacts, Inspect AI ReAct agent, 5 tools, 24 h / 5000 message cap, programmatic artifact grading, 20 runs per model (260 runs), each model at its maximum thinking effort; metric mean task score 0-1; cost = mean USD per task over 20 tasks from per-call tokens x provider list prices; arXiv 2608.25286 (Aug 2026); read 2026-10-20 ✓xmodelGemini 3.5 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Flash-Lite·high
bsb-v1-green
github-bullshitbench-v1
—xmodelGemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·low · Gemini 3 Pro·high · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3.6 Flash·minimal · Gemini 3 Flash·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high
BVB
arxiv-2609.15478
arXiv 2609.15478 · Mini-BVB via LiteLLM, $3/scene cap, 5 Claude models at high (+ omitted-effort runs), LiteLLM list cost ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3 Flash·high
c0ze/agent-evaluation: 13 tasks, blind-graded (24-way pass of 13 Oct)
c0ze-agent-evaluation
13 self-contained tasks, 1 run per model, blind rubric grading by Opus 5.5 subagents; score = mean of 13 task totals /10; cost per 13-task run (credit-balance drop, whole $; Haiku API-equivalent); read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high
cauchye_asia Japanese text + vision, basic set (21 q x 2)
zenn-cauchye_asia
paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium
cauchye_asia Japanese text + vision, hard set (10 q x 2)
zenn-cauchye_asia
paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium
CausalGame Agent mode (structured tool calling): mean survival rate over 14 experiments
causalgame-2607.04293
CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Agent mode (tool calls); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓xmodelGemini 3.5 Flash·medium · Gemini 3.1 Flash-Lite·minimal
CausalGame Prompting mode (Python code): mean survival rate over 14 experiments
causalgame-2607.04293
CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Prompting mode (Python code); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓xmodelGemini 3.5 Flash·medium · Gemini 3.1 Flash-Lite·minimal
CEO-Bench
arxiv-2606.18543
terminal-agent · Opus/Sonnet=MAX, Haiku=thinking ✓xmodelGemini 3.5 Flash·high
Chess puzzle accuracy, base prompt (imec APO paper)
imec-2610.00416
559-puzzle test set, base prompt (no optimisation), mean of 3 runs, generative single-move answer, legal-move list given, no tools; metric puzzle accuracy %; cost = USD per 1,000 solver-position calls; arXiv 2610.00416 Table 2 (Oct 2026); read 2026-10-20 ✓xmodelGemini 3.5 Flash·medium · Gemini 3.5 Flash-Lite·low
CL-Bench
clbench.com
CL-Bench, learning from context · overall score on 0–1 with its std in the flags; Opus 4.5 with and without thinking as default/nothink; no cost; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Pro·high
ClerkBench banded_meter_billing (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload
zenodo-23048151
Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelGemini 3.7 Flash·medium · Gemini 3.7 Flash·high
ClerkBench fee_ledger_dropzero (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload
zenodo-23048151
Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelGemini 3.7 Flash·medium · Gemini 3.7 Flash·high
clerkbench-ch6
zenodo-23048151
—xmodelGemini 3.7 Flash·medium
cliphou.se Motion Graphics Benchmark v1.1: cost per run (Remotion videos)
cliphou.se
3 briefs x 2 runs via OpenRouter, reasoning high; cost = mean OpenRouter generation cost per run over the 6 runs (computed by us); no quality score published (human review pending); read 2026-10-15 ✓xmodelGemini 3.8 Flash·high
Code4Scene private set (95 cases), overall S_model
arxiv-2609.36777
Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; private set (95 cases), overall S_model; read 2026-10-19 ✓xmodelGemini 3.8 Flash·high
Code4Scene public set (95 cases), overall S_model
arxiv-2609.36777
Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; public set (95 cases), overall S_model; read 2026-10-19 ✓xmodelGemini 3.8 Flash·high
CodeActionBench, Reference harness (single code-generation loop)
arxiv-2609.33807
CodeActionBench (arXiv 2609.33807, 27 Sep 2026): 25 tasks x 3 seeds = 75 attempts per configuration, agentic Code-as-Policy for embodied manipulation; 'Reference' harness for all rows of this group; score = Success (/75 attempts); task coverage /25 is another summary; cost = Table 11 total API-equivalent USD over the 75 attempts (computed by the authors from tokens at list rates); read 2026-10-20 ✓xmodelGemini 3.6 Flash·high
Codex Radar intelligence-efficiency board, Codex CLI harness
codexradar
Codex Radar recurring repository-level coding eval (per-task pass over valid tasks; payload mode equal_latest_3: equal-weight latest three valid samples per task); harness codex; snapshot payload stamped 2026-10-01T23:00:27+08:00 read 2026-10-20; score=passed/valid_tasks (derived; payload gives passed, valid_tasks, iq=150*pass_rate); cost=average_price_usd per run (estimated USD-equivalent from token prices); valid_tasks differ per row (36..336); read 2026-10-20 ✓xmodelGemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high
Conceptual Reasoning Index · DTBench
conceptualreasoning.ai
Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓xmodelGemini 4 Argon·low · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 3.8 Flash·medium · Gemini 3.1 Pro·medium · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Flash-Lite·high
Conceptual Reasoning Index · LMCA
conceptualreasoning.ai
Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓xmodelGemini 4 Argon·low · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 3.8 Flash·medium · Gemini 3.1 Pro·medium · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Flash-Lite·high
Convex evals (no guidelines)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
Convex evals (with guidelines)
convex.dev
Convex coding evals, 112 evals in 8 categories, experiment 'with guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
Convex LLM leaderboard v92a60e0b · no guidelines
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
Convex LLM leaderboard v92a60e0b · no guidelines + web
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines + web; read 2026-10-15 ✓xmodelGemini 4 Argon·medium
Convex LLM leaderboard v92a60e0b · with guidelines
convex.dev
Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, with guidelines; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
ctala
github-ctala
Claude Code CLI · reasoning not configured, temp 0.7 ✓xmodelGemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·medium
CWE-bench (120 vulnerability-repair tasks, 4 attempts)
arxiv-2610.17020
Claude Code, high reasoning effort, 4 attempts per task, 1 h limit; programmatic pass@1; cost = mean API cost per rollout; read 2026-10-15 ✓xmodelGemini 4 Argon·high
cwe-bench-v2-class-k01
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k02
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k03
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k04
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k05
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k06
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k07
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k08
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k09
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-class-k10
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-combined-pass1
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-combined-pass4
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-language-c-cpp
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-language-go
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-language-java
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-language-js-ts
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-language-python
cwe-bench
—xmodelGemini 4 Argon·high
cwe-bench-v2-language-rust
cwe-bench
—xmodelGemini 4 Argon·high
DataLLM Lab: 9 executed Python tasks, Qwen3.8 Max family (Prime, Max, Max-0902), pass and cost per 1,000 tasks
datallmlab
9 executed Python tasks (python3 -I subprocess, 12 s timeout), temperature 0, max_tokens 4000, one scored attempt, via OpenRouter; three runs on different days (Prime 2026-10-02, Max 2026-08-20, Max-0902 2026-09-16); cost = measured tokens x list price on the priced-on date (source-computed, 'not a billing statement'); read 2026-10-20 ✓xmodelGemini 3.6 Flash·medium · Gemini 3.1 Pro·medium · Gemini 3.5 Flash·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·medium
datallm-effort-probe
datallmlab
—xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high
DeepSeek V4 preview card: apex
deepseek
apex: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: apexshort
deepseek
apexshort: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: browsecomp
deepseek
browsecomp: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: codeforces
deepseek
codeforces: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: corpusqa
deepseek
corpusqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: csimpleqa
deepseek
csimpleqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: gdpval
deepseek
gdpval: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: gpqa
deepseek
gpqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: hle
deepseek
hle: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: hle-tools
deepseek
hle-tools: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: hmmt26
deepseek
hmmt26: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: imo
deepseek
imo: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: lcb
deepseek
lcb: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: mcpatlas
deepseek
mcpatlas: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: mmlupro
deepseek
mmlupro: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: mrcr
deepseek
mrcr: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: simpleqav
deepseek
simpleqav: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: swepro
deepseek
swepro: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: sweverified
deepseek
sweverified: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: tb20
deepseek
tb20: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSeek V4 preview card: toolathlon
deepseek
toolathlon: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSWE v1.1 (OpenAI GPT-5.6 post chart)
openai
DeepSWE v1.1 pass@1, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
DeepSWE v1.1 (OpenAI GPT-6 Astra post chart, non-code mode)
openai
DeepSWE v1.1; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high
Design Creativity Bench (UI generation; overall T-score, median cost per design)
arxiv-2610.11539
80 UI design tasks / 160 prompts, one generation per design, highest effort (Opus 5.5 at xhigh); Tables 2 and 9 of arXiv 2610.11539; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high
digirise/チャエン: 3 prompts (game/summary/earnings), total time/cost, n=1
digirise-chaen
digirise.ai/chaen-ai-lab/claude-opus-5-5/ (cites チャエン's X post 2026-09-23); Claude Code, 1 run per model per prompt, 3 prompts summed; effort-from-config: "Claudeの2モデルはClaude Code経由でeffortはxhigh"; read 2026-10-01 ✓xmodelGemini 3.8 Flash·high
Dyad app-builder · Deskhero
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Deskhero (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high
Dyad app-builder · Portalis
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Portalis (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high
Dyad app-builder · Relay CRM
github-dyad
dyad-sh/dyad ba27f52 · local-agent mode, Relay CRM (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓xmodelGemini 3.8 Flash·medium · Gemini 3.8 Flash·high
E-Bench (MCP tools): Avg@3 task success
ebench-2607.23722
E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench (MCP tools): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓xmodelGemini 3.5 Flash·high
E-Bench-Code (code execution): Avg@3 task success
ebench-2607.23722
E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench-Code (code execution): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓xmodelGemini 3.5 Flash·high
E-Commerce Bench (arXiv 2608.30730)
arxiv-2608.30730
E-Commerce Bench, simulated e-commerce store, 365-day task, 18 tools, 4000-turn cap, 5 independent episodes per model in one fixed world, harness = the paper's agent loop; score = mean end-of-year total assets (kilo-yuan) from a 100k stake, bankrupt episodes kept in the mean; no cost reported; arXiv v1 2026-08-31; read 2026-10-20 ✓xmodelGemini 3.5 Flash·high · Gemini 3.1 Pro·high
econ-eval leaderboard A (20 tasks, Opus-5 judge)
github-deluair-econ-eval
deluair/econ-eval README at commit 77f9f3e (16 Oct 2026), leaderboard of 2026-09-10: 20 economics tasks (quantitative, reasoning, coding, writing) x 5 samples = 100 completions/model, numeric/exact/code-exec graders + rubric tasks judged by Claude Opus 5; score = mean over tasks of task means; cost = total for the 100 completions from token counts x list price (prices read from the OpenRouter catalog 2026-07-31 / vendor pages); per-track tables are the same completions (not extra groups); ten models ran 2026-08-01, deepseek-flash added 2026-09-10 same tasks/samples/judge; read 2026-10-20 ✓xmodelGemini 3.6 Flash·medium
econ-eval leaderboard B (50 tasks, GPT-6 Astra judge)
github-deluair-econ-eval
deluair/econ-eval README at commit 77f9f3e, leaderboard B: 50 tasks (the 20 above + 30 daily-work tasks added 2026-09-15), graded by GPT-6 Astra; 5 samples per task (Argon 1); runs dated 2026-09-15 / 2026-10-13 / 2026-10-15; cost only where an API price exists; different judge and task set from leaderboard A: separate group; read 2026-10-20 ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
EMB
vals.ai
Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
ENSO stochastic-model building (SciExam), final composite score
arxiv-2610.10513
agent builds an ENSO model within 6 h eligible time, one run per system, Claude Code on subscription; Table 4/6 of arXiv 2610.10513; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium
Epoch · chess puzzles
epoch.ai
Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 3.1 Flash-Lite·minimal · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high · Gemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·minimal · Gemini 3.8 Flash·high · Gemini 3 Flash·high · Gemini 4 Carbon·high
Epoch · ebr bench
epoch.ai
Epoch AI's own evaluation (ebr_bench), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 4 Carbon·high
Epoch · frontiermath erdos
epoch.ai
Epoch AI's own evaluation (frontiermath_erdos), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 4 Carbon·high
Epoch · frontiermath tier 4 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.6 Flash·high · Gemini 4 Carbon·high
Epoch · frontiermath tiers 1 3 v2
epoch.ai
Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high · Gemini 3.1 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 4 Carbon·high
Epoch · furniture assembly
epoch.ai
Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 4 Carbon·high
Epoch · gpqa diamond
epoch.ai
Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·low · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Pro·high · Gemini 3.1 Flash-Lite·minimal · Gemini 3.1 Flash-Lite·high · Gemini 3.1 Flash-Lite·low · Gemini 3 Flash·high · Gemini 3.6 Flash·high · Gemini 4 Carbon·high
Epoch · mirrorcode
epoch.ai
Epoch AI's own evaluation (mirrorcode), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.1 Pro·high
Epoch · mystery game puzzles
epoch.ai
Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·low · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·low · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·low · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 4 Carbon·high
Epoch · otis mock aime 2024 2025
epoch.ai
Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 3.8 Flash·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·low · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Flash-Lite·minimal · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high · Gemini 3 Flash·high · Gemini 3.6 Flash·high
Epoch · simpleqa verified
epoch.ai
Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 4 Carbon·high
epoch-ext-deepresearchbench
epoch.ai
—xmodelGemini 3 Flash·low · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.1 Flash-Lite·low
epoch-ext-surface-evolver-bench
epoch.ai
—xmodelGemini 3.8 Flash·high
EXTRA retroboard-antigravity
github-retrospective-board-eval
new group made by verifier v6 from the g6 report (rows flagged extra-group; no held Claude rows unless stated); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3.1 Pro·low
fcode20pass
cognition
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high
fcode20score
cognition
—xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high · Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
FinancialAuditBench (2609.32835) — 45 audit workpaper tasks, bash-only agent, 8 runs/task
arxiv-2609.32835
Table 8/Table 9 main row, strict pass % and mean cost $/trajectory, 360 trajectories/model; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium
fireworks-dsflash-deepswe
fireworks-blog
—xmodelGemini 3.8 Flash·high
FlavourBench Score (mean of 3 families x 178 tasks)
flavourbench-2608.20574
FlavourBench Table 3 leaderboard, 534 tasks, FB Score = equal-family mean, 2 collection panels pooled, OpenRouter fixed provider (GLM 5.3 direct); arXiv 2608.20574 (20 Aug 2026); family columns Table 10 are same runs (rule 8); read 2026-10-20 ✓xmodelGemini 3.1 Pro·low · Gemini 3.6 Flash·minimal
FrontierFinance T4
arxiv-2608.11683
arXiv 2608.11683 Table 4 · effort omitted, Fable 5 vs Opus 4.8 ✓xmodelGemini 3.6 Flash·medium · Gemini 3.1 Pro·high
FrontierSWE dominance score (GLM-5.2 card, as of 2026/06/16)
zhipu
FrontierSWE, evaluated by Proximal, 1M context 128K output, dominance score as of 2026/06/16; held group frontierswe is mean-reward@5 (other metric and date); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
FrontierSWE v2
frontierswe
Proximal frontierswe.com + Sonnet 5.5 card · 34 ultra-long tasks, max effort, proximus harness, mean reward@5; cost basis undocumented ✓xmodelGemini 3.7 Flash·high · Gemini 4 Carbon·high
GDPval-AA v2 (OpenAI GPT-5.6 post chart)
openai
GDPval-AA v2; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓xmodelGemini 3.5 Flash·high
gdpx-overall
semianalysis
—xmodelGemini 4 Carbon·high · Gemini 4 Argon·high
Gemini 3 Deep Think blog table (12 Feb 2026): cmt
googleblog-deepthink-feb26
blog.google Deep Think post, evaluation table image; cmt; maker-reported, no cost; read 2026-10-20 ✓xmodelGemini 3 Pro·high
Gemini 3 Deep Think blog table (12 Feb 2026): icho2025
googleblog-deepthink-feb26
blog.google Deep Think post, evaluation table image; icho2025; maker-reported, no cost; read 2026-10-20 ✓xmodelGemini 3 Pro·high
Gemini 3 Deep Think blog table (12 Feb 2026): imo2025
googleblog-deepthink-feb26
blog.google Deep Think post, evaluation table image; imo2025; maker-reported, no cost; read 2026-10-20 ✓xmodelGemini 3 Pro·high
Gemini 3 Deep Think blog table (12 Feb 2026): ipho2025
googleblog-deepthink-feb26
blog.google Deep Think post, evaluation table image; ipho2025; maker-reported, no cost; read 2026-10-20 ✓xmodelGemini 3 Pro·high
Google Earth AI GRA paper: CBGB, GRA
googleearthai-2610.15750
CBGB test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: CBGB, LLM
googleearthai-2610.15750
CBGB test set, configuration 'LLM', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: CBGB, LLM + Basic Tools
googleearthai-2610.15750
CBGB test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: CBGB, LLM + GRA Tools
googleearthai-2610.15750
CBGB test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: GeoExpertQueries, GRA
googleearthai-2610.15750
GeoExpertQueries test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: GeoExpertQueries, LLM + Basic Tools
googleearthai-2610.15750
GeoExpertQueries test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: GeoExpertQueries, LLM + GRA Tools
googleearthai-2610.15750
GeoExpertQueries test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: GeoKTools, GRA
googleearthai-2610.15750
GeoKTools test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: GeoKTools, LLM
googleearthai-2610.15750
GeoKTools test set, configuration 'LLM', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: GeoKTools, LLM + Basic Tools
googleearthai-2610.15750
GeoKTools test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: GeoKTools, LLM + GRA Tools
googleearthai-2610.15750
GeoKTools test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: UnivEARTH, GRA
googleearthai-2610.15750
UnivEARTH test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: UnivEARTH, LLM
googleearthai-2610.15750
UnivEARTH test set, configuration 'LLM', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: UnivEARTH, LLM + Basic Tools
googleearthai-2610.15750
UnivEARTH test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google Earth AI GRA paper: UnivEARTH, LLM + GRA Tools
googleearthai-2610.15750
UnivEARTH test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium
Google model cards: APEX-Agents
google
APEX-Agents long horizon professional tasks; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Pro·high
Google model cards: BrowseComp (search + python + browse)
google
BrowseComp, Gemini via Deep Research with search, python and browsing; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Pro·high
Google model cards: GDPval-AA (Elo)
google
GDPval-AA Elo (Artificial Analysis); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Pro·high
Google model cards: MCP Atlas (public set, Turing)
google
MCP Atlas public set sourced from Turing; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Pro·high
Google model cards: SciCode
google
SciCode (Artificial Analysis); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xmodelGemini 3 Pro·high
Google model cards: tau2-bench Retail
google
tau2-bench Retail (Airline excluded); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Google model cards: tau2-bench Telecom
google
tau2-bench Telecom (Airline excluded); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
GRIP closed-loop accuracy (E1, mean over Q1-Q5), spatial and physical reasoning for VLMs
grip-2610.16170
GRIP Table 3 Mean (E1): 34 domains, procedurally generated images with closed-form answers, 5 questions per image in fixed order, fresh single-turn per item, 8,500 queries per hosted model, routed API; hosted max output 256-2048 tokens; Q1-Q5 columns are the same runs (rule 8); arXiv 2610.16170 (Oct 2026); read 2026-10-20 ✓xmodelGemini 3.8 Flash·low
habr-crashtest27
habr-author
—xmodelGemini 4 Argon·medium
Haiku 5.5 system card: biomedical image analysis, max effort
anthropic
CT/MRI/histopathology/dermoscopy/retinal/ultrasound/microscopy imaging, scaled to best public-competition result; 1 attempt/problem (exception to the chapter default); read 2026-10-08 ✓xmodelGemini 3.8 Flash·high
Haiku 5.5 system card: medicinal chemistry (ADME), max effort, xmodel
anthropic
504 questions, drug-like molecule structures, ADME property comparison; read 2026-10-08 ✓xmodelGemini 3.8 Flash·high
Haiku 5.5 system card: morphology-to-molecule matching (Axiom Bio), max effort
anthropic
matches 4 multi-channel cell-imaging sets to 4 shuffled molecule-dose perturbations, human liver in-vitro assay; read 2026-10-08 ✓xmodelGemini 3.8 Flash·high
Haiku 5.5 system card: Protocols, Understanding (V2, Benchling), max effort
anthropic
real lab protocol document + practical application question; read 2026-10-08 ✓xmodelGemini 3.8 Flash·high
Harbor-Index 1.0 (82 tasks), Terminus-2 harness for all models
arxiv-2609.04298
Harbor-Index 1.0: 82 tasks selected from 54 Harbor benchmarks, 3 trials, harness Terminus-2 for every model; pass rate % and approximate per-run cost (USD) of one full Index run; provider default settings (paper: reasoning effort at provider default); models other than GPT-5.5/Opus 4.8/Gemini 3.1 Pro served via OpenRouter; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high
Harbor-Index adapter AA-LCR, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter AA-LCR evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter Aider Polyglot, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Aider Polyglot evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter AIME 24 & 25, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter AIME 24 & 25 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter AlgoTune, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter AlgoTune evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter ARC-AGI-2, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter ARC-AGI-2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter BFCL, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter BFCL evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter BigCodeBench-Hard, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter BigCodeBench-Hard evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter BIX-Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter BIX-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter CodePDE, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CodePDE evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter CompileBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CompileBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter CRUST-Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CRUST-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter CyberGym, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter CyberGym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter DA-Code, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter DA-Code evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter DeepSynth, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter DeepSynth evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter FeatureBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter FeatureBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter FinanceAgent, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter FinanceAgent evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter GAIA, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GAIA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter GAIA2, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GAIA2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter GPQA Diamond, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GPQA Diamond evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter GSO, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter GSO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter HumanEvalFix, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter HumanEvalFix evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3 Flash·high
Harbor-Index adapter Humanity’s Last Exam, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Humanity’s Last Exam evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter IneqMath, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter IneqMath evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter KUMO, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter KUMO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter LAB-Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter LAB-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter LawBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter LawBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter LiveCodeBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter LiveCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter MedAgentBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter MedAgentBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter MMAU, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter MMAU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter MMMLU, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter MMMLU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter Omni-Math, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Omni-Math evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter PIXIU, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter PIXIU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter QCircuitBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter QCircuitBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter QuixBugs, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter QuixBugs evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter Reasoning Gym, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Reasoning Gym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter ReplicationBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter ReplicationBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter ResearchCodeBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter ResearchCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3 Flash·high
Harbor-Index adapter SciCode, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SciCode evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter Seal-0, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Seal-0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SimpleQA, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SimpleQA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SkillsBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SkillsBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SLDBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SLDBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter Spider 2, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Spider 2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SpreadsheetBench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SpreadsheetBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter StrongReject, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter StrongReject evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SWE-Bench Pro, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-Bench Pro evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SWE-bench Verified, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-bench Verified evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SWE-Bench-Multilingual, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-Bench-Multilingual evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SWE-Lancer, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-Lancer evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SWE-smith, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWE-smith evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter SWT Bench, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter SWT Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter Terminal-Bench 2.0, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter Terminal-Bench 2.0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter USACO, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter USACO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
Harbor-Index adapter WideSearch, harness Terminus-2 (3 trials)
arxiv-2609.04298
Harbor benchmark adapter WideSearch evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
HealthBench Professional (OpenAI GPT-6 Astra post chart, non-code mode)
openai
HealthBench Professional; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓xmodelGemini 3.8 Flash·high
hehee9 Maze Bench Tier 1 (30 mazes)
hehee9maze-bcd609c
Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 30 mazes (6 sizes x 5, 4x4..18x18), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status completed); read 2026-10-20 ✓xmodelGemini 3.1 Flash-Lite·medium · Gemini 3.1 Pro·medium · Gemini 3.5 Flash·medium · Gemini 3.5 Flash-Lite·medium · Gemini 3.6 Flash·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·medium
heydonna-rt-v1.1-audio-legal-structure
heydonna
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low · Gemini 3.1 Flash-Lite·minimal
HLE text-only (GLM-5.2 card)
zhipu
HLE text-only subset; HLE, temperature=1.0 top_p=0.95, 163,840 gen tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Humanity's Sixth Sense (HSS), single-pass API, pass@1
arxiv-2610.08966
522 image/video tasks, pass@1 = every rubric criterion met, effort = highest honoured per Table 5; tokens_out = mean thinking + answer tokens per task; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high
Hy-MultiTurn (Chinese deep multi-turn dialogue understanding): Importance score
hymultiturn-2607.29196
Hy-MultiTurn leaderboard (Tencent Hunyuan authors; Hy3/Doubao rows are not in the eight providers): six dimensions, final-turn response scored against weighted rubric items; Importance (weighted correct share) = primary metric, mean of three independent rolls; arXiv 2607.29196 (31 Jul 2026); read 2026-10-20 ✓xmodelGemini 3.1 Pro·high
ibmartian-full-p-gemini
ironbee
—xmodelGemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low
IdentityBench v1: identity-governance decisions (680 test cases), Claude CLI baseline prompt
github-identitybench
rlwillen0121/identity-eval-harness README leaderboard; frozen test split, baseline prompt, provider claude_cli; no effort set; score = grounded accuracy %; cost = API-equivalent USD per 1,000 decisions; read 2026-10-15 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium · Gemini 3.5 Flash-Lite·minimal
IdentityBench v3-Full: 360 multi-turn stateful IGA episodes (max 12 turns), claude_cli provider, 17 Oct 2026
github-identitybench
rlwillen0121/identity-eval-harness README fad1304 22-model v3_full table; experiments/v3-full.yaml; NEW GROUP (held identitybench-v1 is the 680-case single-prompt v1 suite); read 2026-10-18 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
IdentityBench v4.1 v41_lite
github-identitybench
IdentityBench v4.1 v41_lite: 60 enterprise IGA scenarios x 6 governance categories = 360 category tasks, 1,628 required actions, 12-turn cap (T0 single monolith), seed 20261019, 1 run per model, README of 2026-10-20 (commit 56591d0), published 2026-10-19; metric tasks solved of 360; cost = run cost of the whole suite (billed or estimated); read 2026-10-20 ✓xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
identitybench-v41skill-a1
github-identitybench
—xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
identitybench-v41skill-a2
github-identitybench
—xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
identitybench-v41skill-a3
github-identitybench
—xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
identitybench-v41skill-a4
github-identitybench
—xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
ITBench-AA (SRE)
artificialanalysis
Artificial Analysis ITBench-AA (SRE); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.5 Flash·high
K2.5 report Table 4: AdvancedIF
moonshot
AdvancedIF; K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓xmodelGemini 3 Pro·high
K2.5 report Table 4: MMMU (val)
moonshot
MMMU (val); K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓xmodelGemini 3 Pro·high
K2.5 report Table 4: MMVP
moonshot
MMVP; K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓xmodelGemini 3 Pro·high
K2.5 report Table 4: OSWorld-Verified
moonshot
OSWorld-Verified; K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: AIME 2025
moonshot
AIME 2025; avg@32; 96k tokens; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: BrowseComp
moonshot
BrowseComp; no context management (60.6 without vs 74.9 with Discard-all, K2.5 report p.11); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: BrowseComp (w/ctx manage)
moonshot
BrowseComp (w/ctx manage); discard-all context management (K2.5 and DeepSeek-V3.2); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: CharXiv (RQ)
moonshot
CharXiv (RQ); vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: CyberGym
moonshot
CyberGym; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: DeepSearchQA
moonshot
DeepSearchQA; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: FinSearchCompT2&T3
moonshot
FinSearchCompT2&T3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: GPQA-Diamond
moonshot
GPQA-Diamond; avg@8; 96k tokens; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: HLE-Full
moonshot
HLE-Full; full set (text+image); 96k completion budget; HF blocked; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: HMMT 2025 (Feb)
moonshot
HMMT 2025 (Feb); avg@32; 96k tokens; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: IMO-AnswerBench
moonshot
IMO-AnswerBench; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: InfoVQA (val)
moonshot
InfoVQA (val); vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: LiveCodeBench (v6)
moonshot
LiveCodeBench (v6); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: Longbench v2
moonshot
Longbench v2; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: LongVideoBench
moonshot
LongVideoBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: LVBench
moonshot
LVBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: MathVision
moonshot
MathVision; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: MathVista (mini)
moonshot
MathVista (mini); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: MMLU-Pro
moonshot
MMLU-Pro; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: MMMU-Pro
moonshot
MMMU-Pro; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: MotionBench
moonshot
MotionBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: OCRBench
moonshot
OCRBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: OJBench (cpp)
moonshot
OJBench (cpp); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: SciCode
moonshot
SciCode; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: Seal-0
moonshot
Seal-0; avg@4; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: SimpleVQA
moonshot
SimpleVQA; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: VideoMME
moonshot
VideoMME; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: WideSearch (item-f1)
moonshot
WideSearch (item-f1); avg@4; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: ZeroBench
moonshot
ZeroBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.5 card: ZeroBench (w/ tools)
moonshot
ZeroBench (w/ tools); vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓xmodelGemini 3 Pro·high
Kimi K2.6 card: AIME 2026
moonshot
AIME 2026; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: APEX-Agents
moonshot
APEX-Agents; 452 of the 480 public tasks (as Artificial Analysis); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: BabyVision
moonshot
BabyVision; vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: BabyVision (w/ python)
moonshot
BabyVision (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: CharXiv (RQ)
moonshot
CharXiv (RQ); vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: CharXiv (RQ) (w/ python)
moonshot
CharXiv (RQ) (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: Claw Eval (pass^3)
moonshot
Claw Eval (pass^3); Claw Eval v1.1; max tokens per step 16,384; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: DeepSearchQA (f1-score)
moonshot
DeepSearchQA (f1-score); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: HLE-Full
moonshot
HLE-Full; full set; max generation 98,304 tokens; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: HLE-Full (w/ tools)
moonshot
HLE-Full (w/ tools); full set; max generation 98,304 tokens; tools: search/code/browse; per-step 49,152; context management; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: HMMT 2026 (Feb)
moonshot
HMMT 2026 (Feb); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: IMO-AnswerBench
moonshot
IMO-AnswerBench; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: LiveCodeBench (v6)
moonshot
LiveCodeBench (v6); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: MathVision
moonshot
MathVision; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: MathVision (w/ python)
moonshot
MathVision (w/ python); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: MCPMark
moonshot
MCPMark; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: MMMU-Pro
moonshot
MMMU-Pro; vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: MMMU-Pro (w/ python)
moonshot
MMMU-Pro (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: OJBench (python)
moonshot
OJBench (python); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: SciCode
moonshot
SciCode; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: SWE-Bench Multilingual
moonshot
SWE-Bench Multilingual; in-house SWE-agent-derived framework; avg of 10 runs; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: Toolathlon
moonshot
Toolathlon; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Kimi K2.6 card: V* (w/ python)
moonshot
V* (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Labelbox · Implicit Intelligence
labelbox.com
Labelbox leaderboards, Implicit Intelligence · score as printed (integer), top five models only; effort from the label; read 2026-09-30 ✓xmodelGemini 3 Pro·high
lb2-browser-analysis-driven-action
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-browser-constraint-adherence
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh
lb2-browser-data-aggregation
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-browser-sequential-planning
livebench
—xmodelGemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-browser-structural-integrity
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-coding-agent-loop
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
lb2-coding-cloud-infrastructure
livebench
—xmodelGemini 4 Hydrogen·xhigh
lb2-coding-hard-fixes
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-coding-memory-tools
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
lb2-coding-product-ux
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh
lb2-dataanalysis-consecutive-events
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-dataanalysis-table-join
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-dataanalysis-table-reformat
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-design-documents-docx
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-design-pdf
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-design-presentations-ppt
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-longhorizon-delegated-operations
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-longhorizon-finance-compliance-apps
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-longhorizon-long-run-operations
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-longhorizon-operations-commerce-apps
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh
lb2-longhorizon-performance-optimisation
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-longhorizon-systems-engineering
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-reasoning-logic-with-navigation
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-reasoning-spatial
livebench
—xmodelGemini 4 Hydrogen·xhigh
lb2-reasoning-theory-of-mind
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-reasoning-zebra-puzzles
livebench
—xmodelGemini 3.8 Flash·high
lb2-research-legal
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-research-regulatory-compliance
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-research-science-evidence
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-research-strategy
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
lb2-research-tax-finance
livebench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh
leanzero-forge2-medium
leanzero
—xmodelGemini 4 Argon·medium
lechmazur Adversarial LLM Debate Benchmark — Bradley-Terry rating
lechmazur-debate
lechmazur/debate, side-swapped adversarial debates, multi-model judge panel (incl. a Claude judge), Bradley-Terry rating with match counts; read 2026-10-05 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high
lechmazur Creative Writing Benchmark — comparison score
lechmazur-writing
lechmazur/writing, matched creative-brief stories judged pairwise by rotating LLM evaluators, Thurstone/BT-style comparison score centered at 0, bootstrap 95% CI; read 2026-10-05 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high
lechmazur NYT Connections Extended Benchmark — full 940-puzzle board
lechmazur-nyt
lechmazur/nyt-connections, 940 puzzles, quadratic-v1 scoring (g/4)^2 per puzzle, objective (not model-judged); read 2026-10-05 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
lechmazur NYT Connections Extended Benchmark — harder 100-puzzle subset
lechmazur-nyt
lechmazur/nyt-connections, harder 100-puzzle subset, same quadratic-v1 scoring, objective; read 2026-10-05 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
legalforecast-mtd-micro-brier
johnhughes3/LegalForecastBench
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 3.1 Pro·high
LexKairos overall
arxiv-2608.09106
LexKairos: Chinese legal temporal reasoning, 9 sub-tasks; temperature 0, max output 8,000 tokens; Vanilla = direct answer without extra reasoning steps; Thinking Mode = built-in reasoning enabled, thinking effort high; weighted overall; read 2026-10-19 ✓xmodelGemini 3 Flash·high
LiveBench 2.0 · Browser Use suite
livebench
LiveBench 2.0 Browser Use suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high
LiveBench 2.0 · Coding suite
livebench
LiveBench 2.0 Coding suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high
LiveBench 2.0 · Data Analysis suite
livebench
LiveBench 2.0 Data Analysis suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high
LiveBench 2.0 · Design suite
livebench
LiveBench 2.0 Design suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high
LiveBench 2.0 · Long-Horizon suite
livebench
LiveBench 2.0 Long-Horizon suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high
LiveBench 2.0 · Reasoning suite
livebench
LiveBench 2.0 Reasoning suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high
LiveBench 2.0 · Research suite
livebench
LiveBench 2.0 Research suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high
livebench-task-amps_hard
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-code_completion
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-code_generation
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-connections
livebench
—xmodelGemini 3 Flash·minimal
livebench-task-consecutive_events
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-integrals_with_game
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-javascript
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-logic_with_navigation
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-math_comp
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-olympiad
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-paraphrase
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-plot_unscrambling
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-python
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-simplify
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-spatial
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-story_generation
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-summarize
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-tablejoin
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-tablereformat
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-theory_of_mind
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-typescript
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-typos
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
livebench-task-zebra_puzzle
livebench
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high
LLM judge accuracy on the alignment set (439 DPO pairs), original prompts
arxiv-2610.18122
pairwise verdict accuracy vs dataset label, CALM prompt, reasoning off; cost = US$ per 1,000 requests over the full 15,195-request design; read 2026-10-15 ✓xmodelGemini 3.8 Flash·low
LLM judge accuracy on the fact-related set, original prompts
arxiv-2610.18122
pairwise verdict accuracy vs correct answer, CALM prompt, reasoning off; read 2026-10-15 ✓xmodelGemini 3.8 Flash·low
llm_chess — aggregated Elo vs Random + Dragon engine (all Claude models)
llm_chess
maxim-saplin/llm_chess, LLM plays Black vs RANDOM_PLAYER and Dragon engine, Elo anchored to Dragon ladder, 95% CI, average_game_cost as cost per game; read 2026-10-05 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.6 Flash·low · Gemini 4 Argon·high
llm_chess — single 10-game match vs RANDOM_PLAYER (Black win rate)
llm_chess
maxim-saplin/llm_chess raw per-match _aggregate_results.json, n=10 games per model, win% computed by us from black_wins/total_games; read 2026-10-05 ✓xmodelGemini 3.6 Flash·low
llm2014-logic-2026-10-median
llm2014-logic
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high
LLMConfBench
arxiv-2609.20666
arXiv 2609.20666 · conformer energy ranking, 27 molecules × 3 prompts via OpenRouter, Opus 5 and Sonnet 5 at high; cost summed from the repo's per-call logs ✓xmodelGemini 3.6 Flash·high · Gemini 3 Flash·high
LMArena Agent
lmarena.ai
Agent Mode real sessions (Code/Chat/Work) · cost=median $/task (14d window), score=Net Improvement composite signal, effort read from model-variant label; live board, read 27 Sep 2026 ✓xmodelGemini 4 Carbon·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 4 Hydrogen·high · Gemini 3.7 Flash·high
LMArena Document
lmarena.ai
Document arena (Overall, the page's default board: no style control) · blind pairwise votes on answers about uploaded documents, Elo rating, no cost; vote cutoff 13 Sep 2026, so no 5.5 model yet; effort read from the arena's model key (-thinking, -high, -max); live board, read 30 Sep 2026 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.6 Flash·high
LMArena Text
lmarena.ai
Text arena (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-opus-5.5-high-vertex); live board, read 30 Sep 2026 ✓xmodelGemini 4 Argon·high · Gemini 4 Hydrogen·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·medium · Gemini 3 Flash·minimal · Gemini 4 Carbon·high
LMArena Vision
lmarena.ai
Vision arena (Overall, the page's default board: style control) · blind pairwise votes on image-input prompts, Elo rating, no cost; vote cutoff 28 Sep 2026, no 5.5 model yet; effort read from the arena's model key (-thinking, -thinking-16k/32k, -high, -max); live board, read 30 Sep 2026 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.6 Flash·high · Gemini 3 Flash·minimal
LMArena WebDev
lmarena.ai
Code arena, WebDev board (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-sonnet-5-5-high); live board, read 30 Sep 2026 ✓xmodelGemini 4 Carbon·high · Gemini 4 Hydrogen·high · Gemini 4 Argon·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3 Flash·minimal
LMArena WebDev — HTML sub-category
lmarena.ai
lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-html, latest split, publish 2026-10-02; read 2026-10-05 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3 Flash·minimal
LMArena WebDev — React sub-category
lmarena.ai
lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-react, latest split, publish 2026-10-02; read 2026-10-05 ✓xmodelGemini 4 Argon·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3 Flash·minimal
lmcouncil-humanity-s-last-exam
lmcouncil
—xmodelGemini 3.1 Pro·high
M-GATE grammar error detection, MCC mean over 30 languages (temperature 0)
mgate-2608.03803
M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.1; 53 models / 82 configurations March-July 2026; linguist-crafted adversarial grammar items, 30 languages, temperature 0, one run; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓xmodelGemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·minimal
M-GATE round-trip translation, hard subset, normalised score mean over 29 targets
mgate-2608.03803
M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.2; 53 models / 82 configurations March-July 2026; 100 English sources round-tripped into 29 languages, hard subset of 50 sentences, judges GPT-5.4 Mini + Claude Haiku 4.5 + Gemini 3.1 Flash Lite at temp 0; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓xmodelGemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·minimal
MA-Bench mission analysis, benchmark condition, weighted score (GPT-5.6 judge), Opus 5 / Fable 5.1 / Sonnet 5
arxiv-2610.19138
arXiv 2610.19138 Table 4; production judge GPT-5.6; 3000-word cap; 216 briefs (n=30/50/30 per model); read 2026-10-18 ✓xmodelGemini 3.8 Flash·medium · Gemini 3.1 Pro·medium
MathArena · aime 2025
matharena.ai
MathArena, aime--aime_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3 Flash·high
MathArena · aime 2026
matharena.ai
MathArena, aime--aime_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high
MathArena · apex 2025
matharena.ai
MathArena, apex--apex_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high
MathArena · april
matharena.ai
MathArena, arxiv--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high
MathArena · april
matharena.ai
MathArena, arxiv_false--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high
MathArena · august
matharena.ai
MathArena, arxiv--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
MathArena · august
matharena.ai
MathArena, arxiv_false--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
MathArena · brumo 2025
matharena.ai
MathArena, brumo--brumo_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3 Flash·high
MathArena · february
matharena.ai
MathArena, arxiv--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.1 Pro·low
MathArena · february
matharena.ai
MathArena, arxiv_false--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
MathArena · hmmt feb 2025
matharena.ai
MathArena, hmmt--hmmt_feb_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3 Flash·high
MathArena · hmmt feb 2026
matharena.ai
MathArena, hmmt--hmmt_feb_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.5 Flash·high · Gemini 3.1 Pro·high · Gemini 3 Flash·high · Gemini 3.6 Flash·high
MathArena · january
matharena.ai
MathArena, arxiv--january: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.1 Pro·medium · Gemini 3 Flash·high · Gemini 3.1 Pro·low
MathArena · june
matharena.ai
MathArena, arxiv--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high
MathArena · june
matharena.ai
MathArena, arxiv_false--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high
MathArena · june
matharena.ai
MathArena, arxivlean--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high
MathArena · kangaroo 2025 1-2
matharena.ai
MathArena, kangaroo--kangaroo_2025_1-2: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.5 Flash·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high
MathArena · kangaroo 2025 11-12
matharena.ai
MathArena, kangaroo--kangaroo_2025_11-12: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high · Gemini 3.5 Flash·high
MathArena · kangaroo 2025 3-4
matharena.ai
MathArena, kangaroo--kangaroo_2025_3-4: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high
MathArena · kangaroo 2025 5-6
matharena.ai
MathArena, kangaroo--kangaroo_2025_5-6: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 3 Flash·high
MathArena · kangaroo 2025 7-8
matharena.ai
MathArena, kangaroo--kangaroo_2025_7-8: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 3 Flash·high
MathArena · kangaroo 2025 9-10
matharena.ai
MathArena, kangaroo--kangaroo_2025_9-10: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.5 Flash·high · Gemini 3 Flash·high
MathArena · march
matharena.ai
MathArena, arxiv--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
MathArena · march
matharena.ai
MathArena, arxiv_false--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.5 Flash·high · Gemini 3.1 Pro·high
MathArena · may
matharena.ai
MathArena, arxiv--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high
MathArena · may
matharena.ai
MathArena, arxiv_false--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high
MathArena · shortlist 2025
matharena.ai
MathArena, apex--shortlist_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high
MathArena · smt 2025
matharena.ai
MathArena, smt--smt_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓xmodelGemini 3 Flash·high
matharena-arxiv-false-october
matharena.ai
—xmodelGemini 4 Argon·high
matharena-arxiv-false-september
matharena.ai
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
matharena-arxiv-october
matharena.ai
—xmodelGemini 4 Argon·high
matharena-arxiv-september
matharena.ai
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
matharena-arxivlean-september
matharena.ai
—xmodelGemini 4 Argon·high
matharena-leanlean-oct26
matharena.ai
—xmodelGemini 4 Argon·high
MazeBench ASCII no tools
mazebench
MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, no Python tools; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 4 Argon·high
MazeBench ASCII with Python tools
mazebench
MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, Python tools enabled; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓xmodelGemini 3.8 Flash·high
MCP-Atlas public set (GLM-5.2 card)
zhipu
MCP-Atlas 500-task public set, think mode, 10 min timeout, judge Gemini-3.0-Pro; table value 76.8 (chart shows 77.0); read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
MedChem (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelGemini 3.1 Pro·high
MediCrab Lab notes #2: 24 planted errors in one pediatric septic-shock trial
medicrab
same prompt, no web search, 2-3 runs each, graded blind; score = mean errors found of 24; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 4 Argon·medium
Mercor · AdvancedIF
mercor.com
Mercor, AdvancedIF: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high · Gemini 3.6 Flash·high
Mercor · CharXiv
mercor.com
Mercor, CharXiv: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high
Mercor · DeepSWE v1.1
mercor.com
Mercor, DeepSWE v1.1: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high
Mercor · MedXpertQA MM
mercor.com
Mercor, MedXpertQA MM: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high
Mercor · MMLU-Pro
mercor.com
Mercor, MMLU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.1 Pro·high
Mercor · MMMU-Pro
mercor.com
Mercor, MMMU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.7 Flash·high · Gemini 3.5 Flash·high
Mercor · SUPERChem
mercor.com
Mercor, SUPERChem: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.7 Flash·high
Mercor · Terminal-Bench 2.1 Extended
mercor.com
Mercor, Terminal-Bench 2.1 Extended: the public benchmark extended with new expert-built Mercor tasks (99 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.8 Flash·high
Mercor APEX-Accounting
mercor.com
Mercor's own board, APEX-Accounting, harness loop_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Pro·high
Mercor APEX-Agents
mercor.com
Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 4 Argon·high
Mercor APEX-SWE (inspect)
mercor.com
Mercor's own board, APEX-SWE, harness inspect · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3.6 Flash·high
Mercor APEX-v1 (The AI Productivity Index)
mercor.com
Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.7 Flash·high · Gemini 4 Argon·high
mercor-apexacct-mean
mercor.com
—xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Pro·high
mercor-apexv1-mean
mercor.com
—xmodelGemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.7 Flash·high · Gemini 4 Argon·high
Mideind Icelandic LLM leaderboard, Case Government (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_case_marking@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Completion (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_cloze@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Famous Phrases (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_phrases@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Flora & Fauna (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_nature_translations@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Generative Quality (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark generative_quality@v1 primary metric quality_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Icelandic GEC (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_gec@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Icelandic WikiQA (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_wikiqa@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Meanings (Language Proficiency)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_meanings@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Song Lyrics (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_song_lyrics@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mideind Icelandic LLM leaderboard, Textile Terms (Knowledge)
mideind-icelandic-leaderboard
Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_textiles@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓xmodelGemini 4 Argon·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high
Mine Odyssey: Minecraft spatial navigation (success rate, 180 tasks)
arxiv-2610.11328
180 navigation tasks (125 outdoor/55 indoor), 500 decisions or 6 h; Opus 5 at 4 efforts plus Opus 5.5 max; Table 1 of arXiv 2610.11328; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high
MineBench voxel arena
minebench
minebench.ai live API · voxel builds, 15 prompts, Elo arena read 29 Sep, measured cost for 8 of 12 Claude models ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high
Morphology (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelGemini 3.1 Pro·high
MTEB(LLM) overall score (37 tasks) and cost per pass, LLMs prompted as embedders
embedder-2608.12875
MTEB(LLM) 37 tasks (8 classification, 9 clustering, 10 STS, 4 pair-classification, 6 retrieval) run by prompting LLMs; metric mean of category means 0-100; cost = USD for one full pass from API token usage x public rates (March/June 2026), reasoning tokens at output rate; per-category columns are the same runs; Gemini 3 Flash 'low' ablation covers retrieval only (not a group: mean not printed); arXiv 2608.12875 COLM 2026 Tables 1/17; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
neocognition-apprentice-bench-api
neocognition
—xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high
neocognition-apprentice-bench-cua
neocognition
—xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high
NeutronGym McStasBench seen tasks (of 16), one-shot
arxiv-2610.03631
16 seen McStasBench tasks, one episode each, temperature 0; Table 1 of arXiv 2610.03631; read 2026-10-15 ✓xmodelGemini 3.6 Flash·medium · Gemini 3.5 Flash-Lite·minimal
nipun-word-break-micro
nipunbatra.github.io
—xmodelGemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.1 Flash-Lite·minimal · Gemini 3.5 Flash·high
NL2Repo (GLM-5.2 card)
zhipu
NL2Repo temperature=1.0 top_p=1.0 max_new_tokens=48k 400k context, rule+LLM anti-hack; GLM-5.2 card; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
NonoBench
nonobench.com
NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓xmodelGemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high · Gemini 3 Flash·high · Gemini 3 Pro·high
NSMQ Riddles offline exact match (156 riddles)
arxiv-2605.07051
NSMQ Riddles (Ghana quiz science/maths riddles), 2019 set, 156 riddles, offline = all clues at once, one attempt, no tools; temperature 1; open models via Together AI API, closed via LangChain wrapper for the OpenAI API (paper Sec.4); low/high reasoning effort; read 2026-10-19 ✓xmodelGemini 3.1 Pro·low · Gemini 3.1 Pro·high
NSMQ Riddles real-time proxy exact match (156 riddles)
arxiv-2605.07051
NSMQ Riddles 2019 real-time proxy: riddle clues passed in 7-word chunks, model answers and states sure/not sure; 156 riddles; temperature 1; same models/providers as offline; read 2026-10-19 ✓xmodelGemini 3.1 Pro·low · Gemini 3.1 Pro·high
NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections)
matsonj/eval-connections
results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓xmodelGemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3 Flash·minimal · Gemini 3.7 Flash·low · Gemini 3.6 Flash·minimal · Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash·minimal
ockbench-top200
ockbench
—xmodelGemini 3.1 Pro·medium · Gemini 3.1 Pro·low
OpenMC neutron-shielding single-shot script generation (v2): execution gate, L1-L5
zenodo-neutron-shielding
zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) Table §5.1 'the gate' (scripts that ran, of 15 total, L1-L5); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓xmodelGemini 3.8 Flash·medium
OpenMC neutron-shielding single-shot script generation (v2): physics-correct, L2-L4
zenodo-neutron-shielding
zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) §5.2 (physics-correct of 9, L2-L4, the rungs with a scorable physics criterion); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓xmodelGemini 3.8 Flash·medium
OpenRouter VGI-Bench (video general intelligence, 439 multiple-choice questions on long videos): accuracy and cost per question
openrouter-vgi-bench
VGI-Bench public split 439 questions (by Seldon), run through OpenRouter default routing, one turn, full video + question, temperature 0, graded letter after 'Answer:'; listed only with >=395 answered; scores aggregate successful runs of the newest 90 days; board last run 2026-10-18 12:18 UTC; cost = average cost per question as printed; read 2026-10-20 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·medium · Gemini 3.5 Flash-Lite·minimal
OSWorld-Pro
arxiv-2609.24890
arXiv 2609.24890 · OSWorld harness, 250 turns, 4 Claude models all at max, tokens × OpenRouter list incl. cache ✓xmodelGemini 3.8 Flash·high
Parallel Search Capability · with search
parallel.ai
Parallel Search Capability Leaderboard, Search Intelligence Score = mean of DSQA, HLE and WISER (100 questions each) with Parallel Search; cost per task incl. estimated search/extract usage; Claude 5.x at max effort per the page notes (Haiku 5.5 effort unstated); read 2026-10-15 ✓xmodelGemini 4 Argon·high
Parallel Search Capability · without search
parallel.ai
Same suites without the search tool (model alone); inference cost only; read 2026-10-15 ✓xmodelGemini 4 Argon·high
ParseBench document-parsing leaderboard (run-llama/ParseBench)
run-llama/ParseBench
leaderboard.csv at commit afb36bd8 (2026-09-29); Overall score (Tables/Charts/ContentFaithfulness/SemanticFormatting/VisualGrounding average); cost is USD-cents per page from the repo's own pricing table (not independently verified against token counts, so flagged list-price-cost); n (documents) and run count not given in the csv; read 2026-10-01 ✓xmodelGemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.1 Pro·low · Gemini 3.1 Flash-Lite·medium · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·medium
phi-Bench: file-system design and implementation, average pass rate
arxiv-2608.00280
Benchmarking LLMs on File System Design and Implementation (UIUC, arXiv 2608.00280, 4 Aug 2026) Figure 4 'Average' bars: pass rate % over six task types (BU/BI/PM/DE/PO/NF), 5 trials per task averaged; DIGITISED from the figure (src/img/phi_f4_zoom.png, pixel scale from the 60%/80% gridlines, +-1.5 points; script verify/phi_bars.py); Figure 10 labels give DeepSeek-V4-Flash 46 and Gemini-3.1-Pro 64 (different weighting); read 2026-10-20 ✓xmodelGemini 3.1 Pro·high
PM-LLM-Benchmark v3, process-mining Q&A, judged by grok-4.7 (fit-alessandro-berti/pm-llm-benchmark3)
fit-alessandro-berti/pm-llm-benchmark3
leaderboard_grok-4.7.md at commit ab1371c (2026-09-30); 1-shot, 10-category score out of 60, LLM judge (grok-4.7, not a Claude judge); single run; no cost in this board; read 2026-10-01 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high
pokebench.tv: Pokemon Red speedrun to Brock, turns/time/cost
pokebench
fresh-save Pokemon Red, screenshot+RAM state per turn, budget 1000 turns, Anthropic API list-price cost; https://pokebench.tv/runs.json, src/pokebench_runs.json; read 2026-10-05 ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
pokelike llm-bench harness v4 (badges mean, 50 seeds)
pokelike
pokelike llm-bench harness v4 (Pokemon-style game, 50 seeds 10000-10049, 1 pass per model, set notes=12 only); metric badges_mean of 50 runs (0-8 cap); tokens per run from the pass JSON; $/run computed by the README table from published per-token prices at print time; runs 2026-08-21 and 2026-08-22; saved repo pierpierpy/pokelike.xyz.bot; read 2026-10-20 ✓xmodelGemini 3.7 Flash·medium
PostTrainBench (GLM-5.2 card, evaluated by PostTrainBench)
zhipu
PostTrainBench, 1M context 128K output; board evaluation by posttrainbench.com; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
prinzbench (private legal-research + search benchmark by a single author), leaderboard of 18 Jul 2026
github-prinz-ai-prinzbench
prinzbench (github.com/prinz-ai/prinzbench README 'Last updated on July 18, 2026'): 33 author-written questions (25 legal research + 8 needle search) asked 3 times, pass@1, graded by the author without blinding; score = full score x/99; access through chat apps or terminals at different dates (Jan-Jul 2026); '*' = early access before public release; the author cautions that Pro models use parallel test-time compute; read 2026-10-20 ✓xmodelGemini 3 Flash·high · Gemini 3 Pro·high
ProgramDistill-300
arxiv-2609.18805
arXiv 2609.18805 · 300 web-app repair tasks, R2E-Gym harness, Opus 5 vs Sonnet 5 at high, tokens × LiteLLM list ✓xmodelGemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high
ProofBench v1.0-era values (vals.ai, Epoch relay) · accuracy %
epoch.ai (vals.ai ProofBench v1.0-era)
ProofBench v1.0-era values relayed by Epoch AI (proofbench_external.csv, rows annotated 'v1.0-era value; model absent from the v1.1 board ... Not comparable to v1.1 rows'); vals.ai ProofBench v1.0 methodology (the held group valsproof is the v1.1 re-grade of 2026-08-14); cost per test as relayed; read 2026-10-20 ✓xmodelGemini 3.6 Flash·high · Gemini 3 Flash·high · Gemini 3.5 Flash-Lite·high
Protein trouble-shoot (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelGemini 3.1 Pro·high
Protein understanding (S55)
anthropic
Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓xmodelGemini 3.1 Pro·high
rails-orig-located
claude-maintain-rails-evals
—xmodelGemini 4 Argon·high
reactbench
reactbench
—xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·low
reactbench-fix
reactbench
—xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·low
reactbench-write
reactbench
—xmodelGemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·low
Reader-highlight sentence ranking (215 dev pages), zero-shot
arxiv-2610.17768
average-precision difference vs the cached gemini-3.1-pro-preview reference, first sample, one scoring run; low reasoning effort; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium
RealityBench acceptability (1,378 frames)
arxiv-2610.14397
zero-shot VLM acceptability F1 against the rule scorer; cost = whole set at developer list price, thinking at output rate; read 2026-10-15 ✓xmodelGemini 3.8 Flash·medium
Reddit 1x1jbry (AlexXx52): one read-only Unity investigation, subscription allowance
reddit-AlexXx52
same read-only investigation of a Unity 6 project on each consumer subscription (Claude Pro for Claude); 'cost' = % of the subscription's own usage allowance (NOT USD, not comparable across providers); score = author's subjective quality /10; 1 run; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high
ReLE 专业能力 (professional ability)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/专业能力.md (专业能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
ReLE 医疗与心理健康 (medical/mental health)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/医疗与心理健康.md (医疗与心理健康); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
ReLE 总分 (total)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/total.md (alldata); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
ReLE 推理与数学计算 (reasoning/math)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/推理与数学计算.md (推理与数学计算); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
ReLE 教育 (education)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/教育.md (教育); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
ReLE 法律与行政公务 (law/admin)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/法律与行政公务.md (法律与行政公务); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 3.8 Flash·medium · Gemini 4 Argon·medium
ReLE 语言与指令遵从 (language/IF)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/语言与指令遵从.md (语言与指令遵从); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
ReLE 通用能力 (general ability)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/通用能力.md (通用能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
ReLE 金融 (finance)
chinese-llm-benchmark
jeinlee1991/chinese-llm-benchmark leaderboard/金融.md (金融); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓xmodelGemini 4 Argon·medium · Gemini 3.8 Flash·medium
renkon40 · sycophancy test, 20 questions
note-renkon40
note.com 30 Sep · Claude Code 2.1.284, sycophancy test, 20 questions; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓xmodelGemini 3.8 Flash·high
ReplaySCM Block-order (100 problems): HeldoutWorldExact
replayscm-2605.08197
ReplaySCM Table 1 Block-order (100 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓xmodelGemini 3.1 Pro·medium
ReplaySCM Hidden-order Hid-Full (250 problems): HeldoutWorldExact
replayscm-2605.08197
ReplaySCM Table 1 Hidden-order Hid-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓xmodelGemini 3.1 Pro·medium
ReplaySCM Ord-Full (250 problems): HeldoutWorldExact
replayscm-2605.08197
ReplaySCM Table 1 Ord-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓xmodelGemini 3.1 Pro·medium
roblox-leetcode-easy
hf-roblox-llm-leaderboard
—xmodelGemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high
roblox-leetcode-hard
hf-roblox-llm-leaderboard
—xmodelGemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high
roblox-leetcode-medium
hf-roblox-llm-leaderboard
—xmodelGemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high
roblox-qa-freeform
hf-roblox-llm-leaderboard
—xmodelGemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high
roblox-qa-multiple-choice
hf-roblox-llm-leaderboard
—xmodelGemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high
robocolis-agent2act
arxiv-2610.27865
—xmodelGemini 4 Argon·medium
Roboflow Playground Vision Evals, OCR (low tier)
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: OCR; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-ocr) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓xmodelGemini 4 Argon·low · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.6 Flash·low · Gemini 3.1 Pro·low · Gemini 3 Flash·low · Gemini 3.5 Flash-Lite·low
Roboflow Playground Vision Evals, overall (mean of tasks) with cost per sample
roboflow-playground-evals
Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; Overall = page's mean over the tasks scored (4/5 where OCR is missing); cost = 'Avg cost / sample' in USD from price x tokens at the pricing date 2026-10-19 ('~' = estimated from self-host/OpenRouter pricing); read 2026-10-20 ✓xmodelGemini 4 Argon·low · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.6 Flash·low · Gemini 3.1 Pro·low · Gemini 3 Flash·low · Gemini 3.5 Flash-Lite·low
RoboQuest (10 tasks, 50 episodes per cell), Inspect Robots harness
arxiv-2610.10388
200-decision budget per episode, success rate over 50 episodes x 10 tasks; cost per episode at list prices; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high
rt-v1.1-audio-proofread
heydonna
—xmodelGemini 3.8 Flash·low · Gemini 4 Argon·low · Gemini 3.1 Flash-Lite·minimal
Scale EnigmaEval (Wayback 2026-10-16, current 5-model board)
scale-seal
https://labs.scale.com/leaderboard/enigma_eval; Wayback capture 20261016182047 (saved gpt-5.1-chat-latest/src/scale/dec_enigma_eval.html); score = board score % with 95% CI; puzzle hunts; current board (models listed with a -high / -xhigh suffix); read 2026-10-20 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
Scale Humanity's Last Exam text-only (Wayback 2026-10-07)
scale-seal
https://labs.scale.com/leaderboard/humanitys_last_exam_text_only; Wayback capture 20261007135520 (saved gpt-5.1-chat-latest/src/scale/dec_humanitys_last_exam_text_only.html); score = board score % with 95% CI; HLE text-only, board page of 2026-10-07 (HLE-Rolling note of 17 Sep 2026); read 2026-10-20 ✓xmodelGemini 3.1 Pro·high
scf52-autom
anthropic
—xmodelGemini 4 Carbon·high
scf52-briefe
anthropic
—xmodelGemini 4 Carbon·high
scf52-chartn
anthropic
—xmodelGemini 4 Carbon·high
scf52-charto
anthropic
—xmodelGemini 4 Argon·high
scf52-chartr
anthropic
—xmodelGemini 4 Argon·high
scf52-fc20
anthropic
—xmodelGemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
scf52-fswe2
anthropic
—xmodelGemini 4 Carbon·high · Gemini 4 Argon·high
scf52-oswp
anthropic
—xmodelGemini 4 Carbon·high
scf52-robodojo
anthropic
—xmodelGemini 4 Argon·high
scf52-tbsci
anthropic
—xmodelGemini 4 Carbon·high
SCHEME (arXiv 2605.29178) · main-task pass rate, side task removed (benign baseline)
arxiv-2605.29178
SCHEME multi-agent coding benchmark, baseline runs with include_side_task=false: 17 instances x 5 epochs = 85 trajectories per model; temperature 1.0; 32,000 max output tokens; rates printed as rounded integers in the text; read 2026-10-20 ✓xmodelGemini 3.1 Pro·high
scheme-sabotage-success
arxiv-2605.29178
—xmodelGemini 3.1 Pro·high
SenseBench (WSD) · prompt p001
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Flash-Lite·minimal · Gemini 4 Argon·high · Gemini 4 Argon·low
SenseBench (WSD) · prompt p002
sense-bench.com
SenseBench lexEN v1, word-sense disambiguation, prompt set p002 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓xmodelGemini 3 Flash·high · Gemini 3.1 Flash-Lite·minimal
SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies): MCC on primary studies
sesr-eval-2610.10633
SESR-Eval screening of SE systematic reviews (SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies)): LLM title/abstract screening via OpenRouter + Pydantic AI, MCC pooled over primary studies (Accuracy/Precision/Recall/F1/lost evidence and the secondary-study table are other summaries of the same runs: rule 8); cost = USD per secondary study (5% sample) from the secondary-study table; Claude 3.7 Sonnet is an old model whose cost is a previously recorded estimate; arXiv 2610.10633 (Oct 2026); read 2026-10-20 ✓xmodelGemini 3.1 Pro·low
SkillHorizon long-horizon business-agent tasks: task success (overall, all reference-workload bands)
skillhorizon-2610.14966
SkillHorizon Table 1 TS Overall: 7 executable MCP business environments, 5 reference-workload bands 64K-1M tokens, 210 executions per model and band (1,050 per model), Harbor + Inspect, context compaction at 200K, temperature/top-p unspecified; TS = proportion of tasks satisfying all final requirements; LGC (local-goal completion) and per-band columns = same runs (rule 8); arXiv 2610.14966 (Oct 2026); read 2026-10-20 ✓xmodelGemini 3.8 Flash·medium
SOC T1a concentration estimation (MAE g C/kg, lower better)
soc-llm-2610.18201
Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓xmodelGemini 3.8 Flash·low
SOC T1b concentration estimation with extra predictors (MAE g C/kg, lower better)
soc-llm-2610.18201
Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓xmodelGemini 3.8 Flash·low
SOC T2 stock calculation accuracy (%)
soc-llm-2610.18201
Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓xmodelGemini 3.8 Flash·low
SOC T3 pairwise paddock choice accuracy (%)
soc-llm-2610.18201
Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓xmodelGemini 3.8 Flash·low
SpireBench: Slay the Spire Ascension 20 Ironclad, mean final floor reached
arxiv-2610.12596
10 runs per model on the same held-out seeds, themed reskin with note-taking; Table 1 and Table 26 of arXiv 2610.12596; read 2026-10-15 ✓xmodelGemini 3.8 Flash·low · Gemini 3.8 Flash·high
StagedWorkspace APEX-Agents file-editing subset, diffs hidden
stagedws-2608.18050
APEX-Agents 57 file-editing tasks, SW-Agent dual views, review-axis arm: workspace diff tools diffs hidden; metric mean rubric score %; arXiv 2608.18050 Table 3c (Aug 2026); read 2026-10-20 ✓xmodelGemini 3 Flash·high
StagedWorkspace APEX-Agents file-editing subset, diffs visible
stagedws-2608.18050
APEX-Agents 57 file-editing tasks, SW-Agent dual views, review-axis arm: workspace diff tools diffs visible; metric mean rubric score %; arXiv 2608.18050 Table 3c (Aug 2026); read 2026-10-20 ✓xmodelGemini 3 Flash·high
StagedWorkspace APEX-Agents mean rubric score, artifact-only (native files, no parsed search)
stagedws-2608.18050
APEX-Agents 452 tasks (480 minus 28 Investment Banking tasks needing EDGAR), SW-Agent harness, arm artifact-only (native files, no parsed search), 3 reruns per task, grader gemini-3-flash-preview at low thinking; metric mean rubric score % (Pass@1 of Table 2b = same runs); cost = mean agent-run USD per task at public list rates for model and run date, parsing/sandbox excluded (Table A.5); arXiv 2608.18050 Tables 2b/3b/A.5 (Aug 2026); read 2026-10-20 ✓xmodelGemini 3 Flash·high
StagedWorkspace APEX-Agents mean rubric score, dual views (native + parsed)
stagedws-2608.18050
APEX-Agents 452 tasks (480 minus 28 Investment Banking tasks needing EDGAR), SW-Agent harness, arm dual views (native + parsed), 3 reruns per task, grader gemini-3-flash-preview at low thinking; metric mean rubric score % (Pass@1 of Table 2b = same runs); cost = mean agent-run USD per task at public list rates for model and run date, parsing/sandbox excluded (Table A.5); arXiv 2608.18050 Tables 2b/3b/A.5 (Aug 2026); read 2026-10-20 ✓xmodelGemini 3 Flash·high
StagedWorkspace APEX-Agents mean rubric score, parsed-only (hash-synced parsed cache, no native files)
stagedws-2608.18050
APEX-Agents 452 tasks (480 minus 28 Investment Banking tasks needing EDGAR), SW-Agent harness, arm parsed-only (hash-synced parsed cache, no native files), 3 reruns per task, grader gemini-3-flash-preview at low thinking; metric mean rubric score % (Pass@1 of Table 2b = same runs); cost = mean agent-run USD per task at public list rates for model and run date, parsing/sandbox excluded (Table A.5); arXiv 2608.18050 Tables 2b/3b/A.5 (Aug 2026); read 2026-10-20 ✓xmodelGemini 3 Flash·high
SuperCLUE 2025年度测评 代码生成
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
SuperCLUE 2025年度测评 幻觉控制
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
SuperCLUE 2025年度测评 数学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
SuperCLUE 2025年度测评 智能体(任务规划)
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
SuperCLUE 2025年度测评 科学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
SuperCLUE 2025年度测评 精确指令遵循
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
SuperCLUE 2026年3月 代码生成
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
SuperCLUE 2026年3月 幻觉控制
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
SuperCLUE 2026年3月 数学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
SuperCLUE 2026年3月 智能体(任务规划)
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
SuperCLUE 2026年3月 科学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
SuperCLUE 2026年3月 精确指令遵循
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3 Flash·high
SuperCLUE 2026年5月 代码生成
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
SuperCLUE 2026年5月 幻觉控制
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
SuperCLUE 2026年5月 数学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
SuperCLUE 2026年5月 智能体(任务规划)
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
SuperCLUE 2026年5月 科学推理
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
SuperCLUE 2026年5月 精确指令遵循
superclue-general
SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
Surface Evolver bench
yhenon-surface-evolver
yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓xmodelGemini 3.8 Flash·high · Gemini 3.5 Flash·medium
surfevolver-pass
yhenon-surface-evolver
—xmodelGemini 3.8 Flash·high · Gemini 3.5 Flash·medium
Surge AI · Antidote: Everyday Edition
surgehq.ai
Surge AI Antidote: Everyday Edition leaderboard (surgehq.ai/benchmarks/antidote); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·medium · Gemini 3.5 Flash·medium · Gemini 3.7 Flash·medium · Gemini 3.5 Flash-Lite·minimal
Surge AI · Chartography
surgehq.ai
Surge AI Chartography leaderboard (surgehq.ai/benchmarks/chartography); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·minimal
Surge AI · ComplexConstraints
surgehq.ai
Surge AI ComplexConstraints leaderboard (surgehq.ai/benchmarks/complex-constraints); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash-Lite·high
Surge AI · DAYJOB: Finance
surgehq.ai
Surge AI DAYJOB: Finance leaderboard (surgehq.ai/benchmarks/dayjob-finance); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Surge AI · DAYJOB: Healthcare
surgehq.ai
Surge AI DAYJOB: Healthcare leaderboard (surgehq.ai/benchmarks/dayjob-healthcare); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
Surge AI · EnterpriseBench: CoreCraft
surgehq.ai
Surge AI EnterpriseBench: CoreCraft leaderboard (surgehq.ai/benchmarks/enterprisebench-corecraft); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 3 Flash·high · Gemini 3 Pro·high
Surge AI · GDP.pdf
surgehq.ai
Surge AI GDP.pdf leaderboard (surgehq.ai/benchmarks/gdp-pdf); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3 Flash·high
Surge AI · GDP.xlsx
surgehq.ai
Surge AI GDP.xlsx leaderboard (surgehq.ai/benchmarks/gdp-xlsx); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high
Surge AI · HANDBOOK.md Agents
surgehq.ai
Surge AI HANDBOOK.md Agents leaderboard (surgehq.ai/benchmarks/handbook); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 3.5 Flash·high · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high
Surge AI · Hemingway-bench
surgehq.ai
Surge AI Hemingway-bench leaderboard (surgehq.ai/benchmarks/hemingway-bench); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 3.7 Flash·medium · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash·medium · Gemini 3 Pro·high · Gemini 3 Flash·high · Gemini 3.5 Flash-Lite·minimal
Surge AI · Riemann-bench
surgehq.ai
Surge AI Riemann-bench leaderboard (surgehq.ai/benchmarks/riemann-bench); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash-Lite·high
Surge AI · sudo L7
surgehq.ai
Surge AI sudo L7 leaderboard (surgehq.ai/benchmarks/sudo-l7); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓xmodelGemini 4 Argon·high · Gemini 3.8 Flash·high
surge-twi-oct9
surge-twi
—xmodelGemini 3.7 Flash·high · Gemini 3.8 Flash·high
SWE-Marathon (GLM-5.2 card, evaluated by Abundant AI)
zhipu
SWE-Marathon, 1M context 128K output; evaluation by Abundant AI; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
synthbench-althing-globalopinionqa
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal
synthbench-althing-gss
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal
synthbench-althing-opinionsqa
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal
synthbench-althing-subpop
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal
synthbench-plain-globalopinionqa
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low
synthbench-plain-gss
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low
synthbench-plain-opinionsqa
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low
synthbench-plain-subpop
github-synthbench
—xmodelGemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low
tau-bench 2 airline (text) leaderboard
github-sierra-tau2-bench
tau2-bench v1.0.1 airline domain, text modality, pass^1 (%) over 4 trials, user simulator gpt-5.2 low reasoning, standard tau-bench scaffold, submitted by Sierra Research (Feb-May 2026), board head 989d328 of 2026-10-20; read 2026-10-20 ✓xmodelGemini 3 Flash·high · Gemini 3 Pro·high
tau-bench 2 retail (text) leaderboard
github-sierra-tau2-bench
tau2-bench v1.0.1 retail domain, text modality, pass^1 (%) over 4 trials, user simulator gpt-5.2 low reasoning, standard tau-bench scaffold, submitted by Sierra Research (Feb-May 2026), board head 989d328 of 2026-10-20; read 2026-10-20 ✓xmodelGemini 3 Flash·high · Gemini 3 Pro·high
tau-bench 2 telecom (text) leaderboard
github-sierra-tau2-bench
tau2-bench v1.0.1 telecom domain, text modality, pass^1 (%) over 4 trials, user simulator gpt-5.2 low reasoning, standard tau-bench scaffold, submitted by Sierra Research (Feb-May 2026), board head 989d328 of 2026-10-20; read 2026-10-20 ✓xmodelGemini 3 Flash·high · Gemini 3 Pro·high
TaxCalcBench TY25, no web search (column-tax/tax-calc-bench)
column-tax-tax-calc-bench
50 TY25 federal+state tax-return cases, 1 run each; strict-correct-return % is the score (lenient% also in the evidence but not used here per instructions); cost = mean litellm-estimate USD/return (n priced noted when <50); source: taxcalc_ty25_claude_by_effort.txt, re-verified against raw evaluation_result_*.md for opus-5.5/ultrathink (40.00%/50.00%/$1.135, exact match); read 2026-10-01 ✓xmodelGemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·minimal · Gemini 3.6 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·low · Gemini 3.8 Flash·low
TaxCalcBench TY25, with web search (column-tax/tax-calc-bench)
column-tax-tax-calc-bench
Same as taxcalc-ty25-notool but with the web-search tool enabled; read 2026-10-01 ✓xmodelGemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.6 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.6 Flash·minimal
Tinybird SQL benchmark
llm-benchmark.tinybird.live
Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓xmodelGemini 3.6 Flash·medium · Gemini 4 Argon·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·medium · Gemini 3.5 Flash-Lite·minimal
Tool-Decathlon (GLM-5.2 card)
zhipu
Tool-Decathlon official evaluation service, max_token 128K; GLM-5.2 card; read 2026-10-19 ✓xmodelGemini 3.1 Pro·high
Toolathlon-Verified leaderboard (current, 108 tasks)
toolathlon
Toolathlon-Verified (released 30 Jun 2026), 108 tasks, 32 MCP servers / 604 tools + 7 local toolkits, Pass@1 % (+-CI) over 3 runs; rows carry the 'Evaluated by us' badge unless noted; Agent 'Default'; page fetched 2026-10-20 (newest row 2026-08-30); cells identical to held cells (vendor cards/blogs copying the board) dropped; read 2026-10-20 ✓xmodelGemini 3.5 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high
UGI Leaderboard — UGI composite (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, fixed prompt set, composite UGI score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓xmodelGemini 3 Pro·high · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high
UGI Leaderboard — World Model factual recall (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, recipe/geoguessr/weight/music percent-error battery; test dates vary by model; read 2026-10-05 ✓xmodelGemini 3 Pro·high · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high
UGI Leaderboard — Writing quality (all Claude models × efforts)
ugi-leaderboard
DontPlanToEnd/UGI-Leaderboard, fixed prompt set, LLM-judged Writing score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓xmodelGemini 3 Pro·high · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high
Vals AIME
vals.ai
Vals AI · AIME, compute_effort null (effort unverified), board 2026-04-16 ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high
Vals CaseLaw v2
vals.ai
Vals AI · CaseLaw v2, compute_effort null (effort unverified), board 2026-05-04 ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high
Vals Code Migration
vals.ai
Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals CorpFin v2
vals.ai
Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high
Vals CUA-bench
vals.ai
Vals AI · CUA-bench via Claude Code, Opus 5.5, Opus 5 and Fable 5.1 at max — snapshot 27 Sep, re-priced (board 22 Sep) ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
Vals CyberBench
vals.ai
Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓xmodelGemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals Finance Agent v2
vals.ai
Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals GPQA
vals.ai
Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Vals HLAB
vals.ai
Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓xmodelGemini 3 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals Index (29 Sep)
vals.ai
vals.ai Vals Index as recomputed on 29 Sep 2026 (older models dropped), max effort, accuracy and cost per test ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals IOI
vals.ai
Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelGemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals Legal Research
vals.ai
Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high
Vals LegalBench
vals.ai
Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals LiveCodeBench
vals.ai
Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Vals MedCode
vals.ai
Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals MedQA
vals.ai
Vals AI · MedQA, Sonnet 4.6 vs Haiku 4.5 thinking, effort null, board 2026-04-16 ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high
Vals MedScribe
vals.ai
Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals MMLU-Pro
vals.ai
Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Vals MMMU
vals.ai
Vals AI · MMMU, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Vals MortgageTax
vals.ai
Vals AI · MortgageTax, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Vals MysteryMechanism
vals.ai
Vals AI · MysteryMechanism, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals ProgramBench
vals.ai
Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓xmodelGemini 3 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals ProofBench
vals.ai
Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals Public Benefits v1
vals.ai
Vals AI · Public Benefits Bench v1 (board 2026-06-09), max + Haiku default ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high
Vals Public Benefits v1.1
vals.ai
Vals AI · Public Benefits v1.1, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelGemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals SAGE
vals.ai
Vals AI · grading student math work against a rubric, all at max ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals Science Index (biology data investigation, mechanism discovery, research workflows)
vals.ai
vals.ai Science Index, equal-weighted BioMysteryBench / MysteryMechanism / Terminal-Bench-Science; cost per test; read 2026-10-18 ✓xmodelGemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals SRE-Bench
vals.ai
Vals AI · binary reverse engineering (despite the name), all at max — snapshot 27 Sep, re-priced ✓xmodelGemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
Vals SWE-bench
vals.ai
Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Vals Tax Agent
vals.ai
Vals AI · Tax Agent, 10 Claude entries at their stated compute_effort — snapshot 27 Sep, re-priced ✓xmodelGemini 4 Carbon·high
Vals TaxEval v2
vals.ai
Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high
Vals TB 4.0
vals.ai
Vals AI · Terminal-Bench 4.0, all Claude at max, measured cost/test — snapshot 27 Sep, re-priced ✓xmodelGemini 4 Carbon·high
Vals Time Horizon KSP
vals.ai
Vals AI · Time Horizon Index: KSP, 3 Claude at max, board 2026-09-14 ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
Vals Vibe Code Bench 1-100
vals.ai
Vals AI · Vibe Code Bench 1-100, Claude models at their stated compute_effort, measured cost/test ✓xmodelGemini 4 Carbon·high
vals.ai Robo-bench (robot arm + humanoid control in a physics sim, most tasks held out)
vals.ai
claude-code harness; overall score; cost = vals 'Est. API cost' per test (USD, estimate); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓xmodelGemini 4 Argon·high
vals.ai SWE-bench Multimodal (JS front-end issues with screenshots)
vals.ai
Vals-run agent; resolve rate overall, cost per test (USD); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
valscodemig11
vals.ai
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high
valsemb11
vals.ai
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high
valsfab21
vals.ai
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high
valsfinindex
vals.ai
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high
valshlab-2026-10-21
vals.ai
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high
valsindex4
vals.ai
—xmodelGemini 4 Carbon·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
valslegalindex
vals.ai
—xmodelGemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high
valssage11
vals.ai
—xmodelGemini 3.5 Flash-Lite·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high
valstaxwp
vals.ai
—xmodelGemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high
vibe-openscad iter-png-1
github-vibeopenscad
vibe-openscad · 7 tasks, 1 PNG-feedback round, 5 Claude models ✓xmodelGemini 3.1 Flash-Lite·minimal
vibe-openscad iter-png-2
github-vibeopenscad
vibe-openscad · 7 tasks, 2 PNG rounds ✓xmodelGemini 3.1 Flash-Lite·minimal
vibe-openscad iter-png-3
github-vibeopenscad
vibe-openscad · 7 tasks, 3 PNG rounds ✓xmodelGemini 3.1 Flash-Lite·minimal
vibe-openscad PDF page
github-vibeopenscad
vibe-openscad · 2 tier-4 datasheet tasks, 8 Claude models default ✓xmodelGemini 3.1 Flash-Lite·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash-Lite·minimal
Video2World (222 instances), coding agents
arxiv-2610.04432
Video2World (arXiv 2610.04432, 3 Oct 2026): 222 instances (189 clips, 39 task families), agent builds a simulation package from video; score = V2WScore (0-100, first column of main table; build rate, functionality success, progress, geometry errors are other columns); seven models in OpenCode (via OpenRouter), GPT-6 Astra in Codex CLI, Fable-5.1 in Claude Code; cost = Table A17 mean API cost per episode (USD); read 2026-10-20 ✓xmodelGemini 3.8 Flash·high
vlm-exam counting
github-vlmexam
roboflow/vlm-exam · 74 samples, LLM-judge accuracy ✓xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · Gemini 4 Argon·low · Gemini 4 Argon·high
vlm-exam detection
github-vlmexam
roboflow/vlm-exam · 250 images, mAP@50 ✓xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · Gemini 4 Argon·low · Gemini 4 Argon·high
vlm-exam extraction
github-vlmexam
roboflow/vlm-exam · 97 samples, LLM-judge accuracy; same models as OCR ✓xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · Gemini 4 Argon·low · Gemini 4 Argon·high
vlm-exam identification
github-vlmexam
roboflow/vlm-exam · 32 samples, near ceiling ✓xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · Gemini 4 Argon·low · Gemini 4 Argon·high
vlm-exam OCR
github-vlmexam
roboflow/vlm-exam · 37 samples, similarity; Fable 5.1 & Opus 5.5 low/high ×3 runs, others low ×1 ✓xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · Gemini 4 Argon·low · Gemini 4 Argon·high
vlmexam-text
github-vlmexam
—xmodelGemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·high · Gemini 3 Flash·low · Gemini 3 Flash·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high · Gemini 4 Argon·low · Gemini 4 Argon·high
WeirdML v3 (agentic, token-axis area score)
epoch-archive
11 ML tasks x 15 configurations; score = token-axis area 0-1 with 95% CI; cost = mean API $/run; agent claude_code; read 2026-10-01 ✓xmodelGemini 3.8 Flash·high
youkiti-prisma-tsuge10-accuracy
github-youkiti
—xmodelGemini 3.8 Flash·medium · Gemini 3 Flash·low · Gemini 3 Pro·high
zapierab-domain-finance
zapier
—xmodelGemini 4 Carbon·medium
zapierab-domain-hr
zapier
—xmodelGemini 4 Argon·high
zapierab-domain-marketing
zapier
—xmodelGemini 4 Carbon·medium
zapierab-domain-operations
zapier
—xmodelGemini 4 Carbon·high · Gemini 4 Argon·medium
zapierab-domain-sales
zapier
—xmodelGemini 4 Argon·medium · Gemini 4 Carbon·high
zapierab-domain-support
zapier
—xmodelGemini 4 Carbon·medium · Gemini 4 Argon·high
zcarbon-domain-finance
zapier
—xmodelGemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
zcarbon-domain-hr
zapier
—xmodelGemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
zcarbon-domain-marketing
zapier
—xmodelGemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
zcarbon-domain-operations
zapier
—xmodelGemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
zcarbon-domain-sales
zapier
—xmodelGemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
zcarbon-domain-support
zapier
—xmodelGemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high
How the numbers are built — the method

Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.

In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.

The procedure

  1. Every number traced to its source. Each source is recorded as facts: the value as the source writes it, the exact text or table cell it is read from, the URL and the date of the capture; agents read every value line by line, and a measurement whose facts lack their proof stays out. Registries of models (the names each was served under, its effort settings and defaults, dated), routes (makers' APIs, resellers, routers, harnesses), benchmarks and their versions, publishers and dated prices let one build decide, by a fixed decision tree, which model, configuration and price each measurement belongs to.
  2. Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale, one version of a board; a group that mixes them is split. Every run is dated, and its model and effort are read as the maker and every route in between served them on that date — never assumed: a route the source does not name is never presumed to be the maker's API, and every route that could have served the run must give the same configuration. At the least doubt on the model, the effort or what was run, the measurement stays in the data file and out of the fit.
  3. Costs at today's prices. Every cost is put on the model's current price from the prices in force on its run date: the source's own figure at that price, the price change applied to the run's own token counts, or, when the token mix is unknown, a price band whose width is added to the cost's precision. A cost that cannot be brought to the current price stays out.
  4. Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The scale is read from the metric's label, never from the benchmark's identity; whether lower is better is read from the benchmark's registry, and such a score is turned in its own unit first (an attack success rate of 30 % reads 70 %); a score whose direction is not established stays out. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
  5. One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result. The sizes of these effects are estimated for each model maker on its own, not tied to the other makers'; a maker's own charts about its own models carry an estimated home effect, also each maker's own, left out of every value. The fit takes the couples of eight makers measured in the same groups (Anthropic, OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi), on one scale; this page shows the Google couples. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
  6. Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source; the one weight set by rule is that of a single run: when a source gives k quality measurements of one run (several metrics, sub-scores or slices of the same executions), each weighs 1/k, so the run counts as one measurement.
  7. Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, a board copying another's numbers, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. Versions of one board that re-grade the same runs (recognised by identical costs) count those runs once. Before the fit, two groups that still share identical costs on three couples or more stop it until the copy is resolved or the coincidence proven from the sources. An index keeps, for a couple whose index components are already in the data, only what its missing components add, or is left out.
  8. Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
  9. From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.

Key values

  • Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
  • Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
  • Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit two figures on this page can move by one unit in that digit (the cost of Gemini 4 Carbon at high, the cost of Gemini 4 Carbon at low), far inside their intervals.
  • Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.

Techniques and software

  • Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes, six chains whose draws continue in the same run until the criteria are met (CmdStan only to resume a stopped fit from its stored draws, or to restart a chain left in a far region from a good chain's last draw); built and checked stage by stage following the Bayesian workflow.
  • Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
  • Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
  • Scales: the empirical logit of bounded scores, with the binomial shape of its noise.

Checks (on the fit of 29 September 2026, before the other makers' couples joined the data). Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.

Alexandre Gensse · github.com/alexandregensse-blip

All figures are indicative and for informational purposes only — derived from public third-party measurements, not an official benchmark, and not affiliated with or endorsed by any model maker. Verify before relying on any value.

Data and text under CC BY 4.0 · code under MIT · download the data (CSV)