Quality vs. cost of every xAI model, at each effort level
One point per couple: a model at one configuration (thinking off, minimal, low → max, thinking on); no line joins a model's couples. Only the couples of the Pareto frontier are labelled (model · effort), where there is room; hover or tap any point for its model, effort, cost and score. Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward.
Best xAI model for the price — the Pareto frontier
The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated; the frontier couples are labelled (model · effort) where there is room — hover or tap any point for its model, effort, cost and score. No line joins a model's couples. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.
Value — distance to the price trend
The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 4 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.84, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.
| Model · effort | cost | expected score | value index trend = 100 |
|---|---|---|---|
| Grok 4.6 · Medium | 2.6× | 62.2 % | 178 |
| Grok 4.6 · Low | 1.0× | 56.2 % | 148 |
| Grok 4.6 · xHigh | 3.3× | 62.3 % | 144 |
| Grok 4.7 · Low | 2.2× | 59.2 % | 118 |
| Grok 4.7 · xHigh | 7.0× | 65.0 % | 112 |
| Grok 4.7 · High | 6.4× | 62.8 % | 80 |
Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.
Best value by task tier — what to actually run
Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.
Picked as the frontier couple that sits furthest below the price trend : it costs 1.8× less than the trend charges for its quality — or, read on the other axis, it scores 3.1 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.
Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — xAI effort levels compared
Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.
| Model | thinking off | low | medium | high | xhigh | max | thinking on | Expected score top effort |
|---|---|---|---|---|---|---|---|---|
| Grok 4.7 | — | 2.21.9–2.5 | 4.43.8–5.0 | 6.45.8–7.2 | 7.06.2–8.0 | — | — | 65.0 % |
| Grok 4.6 | — | 1.00.91–1.1 | 2.62.3–2.8 | 3.02.7–3.2 | 3.32.9–3.6 | — | — | 62.3 % |
"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. A model measured at a single setting (solo : no effort parameter, see the measurement sources) fills one merged cell. A thinking off column appears when a model is measured with its thinking switched off, a minimal one for the lowest level some models have below low; each is a configuration of its own, read from its own measurements. n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).
Value-index matrix — every model and effort against the price trend
The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.
| Model | thinking off | low | medium | high | xhigh | max | thinking on |
|---|---|---|---|---|---|---|---|
| Grok 4.7 | — | 11881–172 | 10570–159 | 8062–103 | 11288–141 | — | — |
| Grok 4.6 | — | 148110–201 | 178132–238 | 9677–119 | 144109–190 | — | — |
Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.
What moves everything — four factors behind the numbers
The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.
Task type
Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.
Complexity
The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).
Generation
Each release buys the same quality for less: turn on Older models to set each family's earlier models beside its current one, on the same scales.
Cache & harness
Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.
Data Sources
Every source below measured at least two (model, effort) couples on the same task; 10115 cost and 23548 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.
| Source | Actual config (harness · effort) | Type | Linked couples (same task) |
|---|---|---|---|
| AA CritPt artificialanalysis | AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Grok 4.1 Fast·think · Grok 4.3·medium · Grok 4.3·low · Grok 4.20·think |
| AA GDP-PDF artificialanalysis | AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓ | sweep | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low |
| AA GDPval v2 artificialanalysis | AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓ | sweep | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh |
| AA Harvey LAB artificialanalysis | AA model payload · Harvey's Legal Agent Bench (AA's own run, separate from Vals's HLAB), score-only, 8 models × ladder + 4 at max ✓ | sweep | Grok 4.3·high · Grok 4.5·high |
| AA HLE artificialanalysis | AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓ | sweep | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·medium · Grok 4.3·low · Grok 4.20·think |
| AA MLCR artificialanalysis | AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓ | sweep | Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| AA Omniscience artificialanalysis | AA model payload · Omniscience accuracy (not the penalised index) ✓ | sweep | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh · Grok 4.20·think · Grok 4.20·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink |
| AA SciCode artificialanalysis | AA model payload · SciCode, 4 models × ladder + 3 at max ✓ | sweep | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low |
| AA TB 4.0 artificialanalysis | AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low |
| AA Terminal-Bench-Science artificialanalysis | AA model payload · Terminal-Bench-Science 0.1 (AA harness, separate from Vals's board), score-only, 4 models × ladder + 2 at max ✓ | sweep | Grok 4.7·xhigh · Grok 4.6·high · Grok 4.7·high · Grok 4.7·low |
| AA-Briefcase artificialanalysis | AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓ | sweep | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low |
| AA-LCR artificialanalysis | AA model payload · long-context reasoning, input-dominated cost ✓ | sweep | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·medium · Grok 4.3·low |
| Agents on Rails · features rubyonrails | rubyonrails.org/ai + rails/ai-evals runs · 20 tickets ×3, miniswen; Opus 5 & Fable 5.1 high→max, Opus 5.5 default ✓ | sweep | Grok 4.6·xhigh · Grok 4.6·high |
| ARC-AGI-1 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓ | sweep | Grok 4.20·think · Grok 4.5·high · Grok 4.5·medium · Grok 4.5·low · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low |
| ARC-AGI-2 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓ | sweep | Grok 4.20·think · Grok 4.5·high · Grok 4.5·medium · Grok 4.5·low · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low |
| bug-hunt-bench github-bughuntbench | Claude Code 2.1.251 · 105 planted bugs, blind judge, 3 models × ladder ✓ | sweep | Grok 4.5·high · Grok 4.6·xhigh · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low |
| BullshitBench v2 github-bullshitbench | petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓ | sweep | Grok 4.6·low · Grok 4.6·xhigh · Grok 4.5·low · Grok 4.5·high · Grok 4.3·low · Grok 4.3·high · Grok 4.1 Fast·nothink |
| CursorBench 3.2 cursorbench | Cursor 3.2 · matched effort, $ cost ✓ (vendor benchmark) | sweep | Grok 4.6·xhigh · Grok 4.6·medium · Grok 4.6·low · Grok 4.6·high · Grok 4.5·high |
| CursorBench 4.0 cursorbench | cursor.com payload · v4.0 long-horizon suite, Opus 5/5.5, Fable 5.1, Sonnet 5/5.5 × low→max, measured tokens × list incl. cache ✓ | sweep | Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·medium · Grok 4.7·low · Grok 4.7·high |
| FrontierCode main cognition | Cognition leaderboard JSON · v1.1 main (100 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓ | sweep | Grok 4.5·low · Grok 4.5·medium · Grok 4.5·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·high · Grok 4.7·xhigh |
| FrontierCode-Ext cognition | Cognition leaderboard JSON · v1.1 Extended (150 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓ | sweep | Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·high · Grok 4.7·xhigh |
| ObviousBench 0.2 benchmarklist | inspect-ai · 144 items ×3 epochs, 3 models × full ladder, measured tokens ✓ | sweep | Grok 4.5·low · Grok 4.5·medium · Grok 4.5·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high |
| PetriBench arxiv-2609.19883 | arXiv 2609.19883 · 4,800 Petri-net questions, raw API adaptive thinking + output_config.effort, Opus 5 M/H/XH + Sonnet 5 M/H, measured tokens × list ✓ | sweep | Grok 4.6·xhigh · Grok 4.6·high · Grok 4.6·medium · Grok 4.5·high · Grok 4.5·medium · Grok 4.3·medium |
| Playcode MacBook SVG playcode | Playcode · one-shot MacBook SVG, high/xhigh/max per model, provider-billed cost (quality ordinal, not scored) ✓ | sweep | Grok 4.6·high |
| Ramen Bench github-ramenbench | not-stbenjam/ramen-bench · one-shot creative-coding HTML, Sonnet 5.5, Opus 5.5, Opus 5, Fable 5.1 × low→max, Claude Code run.json, cost only ✓ | sweep | Grok 4.5·high · Grok 4.5·low · Grok 4.5·medium · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·high · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·xhigh |
| Vals Web Search (finance) vals.ai | vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓ | sweep | Grok 4.5·high |
| Vals Web Search (legal) vals.ai | vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓ | sweep | Grok 4.5·high |
| vlm-exam github-vlmexam | roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓ | sweep | Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high |
| WeirdML epoch-archive | Epoch archive · 3 models × high/max, cost per run ✓ | sweep | Grok 4.6·high |
| FeatherBench ed-is-ai/featherbench | OpenRouter(Anthropic) · 28-task multi-domain suite (coding/data/realworld/security/tool-use), effort=high (Haiku=solo), mean cost/tokens per task-run, consolidated results/summary.json; only Opus 5.5 rung saturated ✓ | xgen | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| Google model cards: FACTS Benchmark Suite | FACTS Benchmark Suite; Google model-card table(s) Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: Global PIQA | Global PIQA; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: Humanity's Last Exam (full set, no tools) | HLE full set text+MM, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: MMMLU | MMMLU multilingual MMLU; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: MMMU-Pro (avg Standard + Vision, no tools) | MMMU-Pro averaged over Standard (10 options) and Vision, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: MRCR v2 8-needle 128k (average) | MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: MRCR v2 8-needle 1M (pointwise) | MRCR v2 8-needle, 1M pointwise; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: SimpleQA Verified | SimpleQA Verified; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Google model cards: SWE-bench Verified (single attempt) | SWE-bench Verified, single attempt; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think |
| Official tau2-bench leaderboard, banking_knowledge (AllTools), Claude submissions by Sierra github-sierra-tau2-bench | banking_knowledge domain, 4 trials, user simulator gpt-5.2 (low), reasoning_effort max for all; pass^1 %; read 2026-10-18 ✓ | xgen | Grok 4.5·high |
| WildClaw arxiv-2605.10912 | OpenRouter (4 harness) · thinking NS ✓ | xgen | Grok 4.20·think |
| Writing-Elo effort/cost chart, Reddit 1wrkwp0 (r/LLMDevs) reddit-1wrkwp0 | same 'Writing Elo' metric family, 10 real YouTube scripts per OP, scored blind by three judges and a rubric; cost per script, log-x scatter with effort-labelled points; no board date printed on the chart; a different Reddit post/source snapshot than 1wr03jc, kept as its own board version; read 2026-10-05 ✓ | xgen | Grok 4.7·medium · Grok 4.7·high |
| 34bench, single Thai bus-route scan transcription, 5 runs/model (dtinth/34bench) dtinth/34bench | results.svg (table-in-SVG, parsed as text) for median accuracy (1-weighted-CER over header/forward/return/footer); meta/ | xmodel | Grok 4.7·high |
| AA AutomationB. artificialanalysis | AA evaluation page · 657 tasks partial-credit, 3 models at max ✓ | xmodel | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low |
| AA Briefcase v1.1 (xAI Grok 4.7 post) xai | AA Briefcase v1.1 Elo, multi-hour office work; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high |
| AA FrontierCode 1.1 (AA-run, weighted score) artificialanalysis | Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low |
| AA MMMU-Pro artificialanalysis | Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Grok 4.20·think · Grok 4.20·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.7·low · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink |
| AA-AnalystAgent artificialanalysis | Artificial Analysis AA-AnalystAgent; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Grok 4.7·high |
| AA-Briefcase (xAI Grok 4.6 post) xai | AA-Briefcase Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| aa-index411 tabbit.ai | — | xmodel | Grok 4.6·high |
| aa-index5 artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| aa5-aa-briefcase artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| aa5-code2video-aa artificialanalysis | — | xmodel | Grok 4.7·xhigh |
| aa5-critpt-v2-hard artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| aa5-frontiercode-2-0 artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| aa5-harvey-lab-aa artificialanalysis | — | xmodel | Grok 4.7·xhigh |
| aa5-healthbench artificialanalysis | — | xmodel | Grok 4.7·xhigh |
| aa5-healthbench-professional artificialanalysis | — | xmodel | Grok 4.7·xhigh |
| aa5-hle-diamond artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| aa5-mlcr-aa artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| aa5-omniscience-non-hallucination artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| aa5-vials-aa artificialanalysis | — | xmodel | Grok 4.7·xhigh |
| aaapex artificialanalysis | — | xmodel | Grok 4.20·think · Grok 4.3·high |
| aaenterpriseops artificialanalysis | — | xmodel | Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| aagpqa artificialanalysis | — | xmodel | Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.20·think |
| aaifbench artificialanalysis | — | xmodel | Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink |
| aaime25 artificialanalysis | — | xmodel | Grok 4.1 Fast·think · Grok 4.1 Fast·nothink |
| aalcb artificialanalysis | — | xmodel | Grok 4.1 Fast·think · Grok 4.1 Fast·nothink |
| aaomni-index artificialanalysis | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh · Grok 4.20·think · Grok 4.20·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink |
| aatau-banking artificialanalysis | — | xmodel | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh |
| aatau2 artificialanalysis | — | xmodel | Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink |
| aatb21 artificialanalysis | — | xmodel | Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh |
| aatbhard artificialanalysis | — | xmodel | Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink |
| Agent Island social-deduction arena (Plackett-Luce skill), OpenRouter models arxiv-2605.04312 | Agent Island (arXiv 2605.04312, 5 May 2026): 999 multi-agent social-deduction games with 49 models via OpenRouter, Bayesian Plackett-Luce posterior skill (mean of lambda; prior mean 1; Table 1); reasoning effort not stated; no cost; read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| AI Coding Daily LLM Coding Leaderboard (4 projects, max 70) aicodingdaily | Laravel 20 + React-TS 20 + bug finding 10 + edge-case 20 = 70; Claude Code rows; author-scored; page 'Last update: October 17th, 2026'; read 2026-10-18 ✓ | xmodel | Grok 4.7·high · Grok 4.6·high |
| aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high) aibenchy | aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · Grok 4.5·medium · Grok 4.6·medium · Grok 4.5·low · Grok 4.7·low · Grok 4.7·medium · Grok 4.6·low · Grok 4.3·medium |
| ALE-Bench epoch-archive | Epoch archive · 3 models at matched high, measured token splits ✓ | xmodel | Grok 4.20·think · Grok 4.6·xhigh · Grok 4.5·high |
| APEX-Agents (xAI Grok 4.6 post) xai | APEX-Agents %, harness not stated (Mercor's benchmark by name; page does not attribute); xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| ARC-AGI-3 (semi-private) arcprize | ARC Prize ARC-AGI-3 semi-private interactive-game set, score in % (as published x100), cost = total run USD; Haiku 5.5 on the Standard harness (the Provider-Adapter harness entry, 2.82 % / $437.14, is a different configuration and is not mixed in); read 2026-10-15 ✓ | xmodel | Grok 4.5·high · Grok 4.5·medium · Grok 4.5·low · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low |
| ArtifactArena model leaderboard, Sampling Harness (SH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| ArtifactArena model leaderboard, Verifier-Grounded Refinement Harness (VGH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| arxiv-2610.12596-spirebench-acts arxiv-2610.12596 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2610.14397-realitybench-doc arxiv-2610.14397 | — | xmodel | Grok 4.6·high |
| arxiv-2610.41025-argon-autopsy-fixrate arxiv-2610.41025 | — | xmodel | Grok 4.7·xhigh |
| arxiv-2610.41076-baba-difficulty-hard arxiv-2610.41076 | — | xmodel | Grok 4.6·high |
| arxiv-2610.41076-baba-difficulty-light arxiv-2610.41076 | — | xmodel | Grok 4.6·high |
| arxiv-2610.41076-baba-difficulty-standard arxiv-2610.41076 | — | xmodel | Grok 4.6·high |
| arxiv-2610.41076-baba-issue-completion arxiv-2610.41076 | — | xmodel | Grok 4.6·high |
| arxiv-2610.41076-baba-pass1-strict arxiv-2610.41076 | — | xmodel | Grok 4.6·high |
| arxiv-2610.41076-baba-pass3 arxiv-2610.41076 | — | xmodel | Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-baseball-elo arxiv-2611.05147 | — | xmodel | Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-football-elo arxiv-2611.05147 | — | xmodel | Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-hockey-elo arxiv-2611.05147 | — | xmodel | Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-nba-elo arxiv-2611.05147 | — | xmodel | Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-nba-gi arxiv-2611.05147 | — | xmodel | Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.07461-t13-grokbuild-baseline-resolution arxiv-2611.07461 | — | xmodel | Grok 4.6·high · Grok 4.5·high |
| arxiv-2611.07461-t13-hermes-resolution arxiv-2611.07461 | — | xmodel | Grok 4.6·high · Grok 4.5·high |
| arxiv-2611.07667-auction-coarse arxiv-2611.07667 | — | xmodel | Grok 4.7·low |
| arxiv-2611.07667-auction-full arxiv-2611.07667 | — | xmodel | Grok 4.7·low |
| arxiv-2611.07667-bargaining-coarse arxiv-2611.07667 | — | xmodel | Grok 4.7·low |
| arxiv-2611.07667-bargaining-full arxiv-2611.07667 | — | xmodel | Grok 4.7·low |
| arxiv-2611.07962-tcsexam-full-constructions arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-index arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-open-conjectures arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-solver arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-theorems arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-constructions arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-index arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-open-conjectures arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-solver arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-theorems arxiv-2611.07962 | — | xmodel | Grok 4.6·xhigh |
| benchlm-aa-agentic-index benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-aa-analystagent benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-coding-index benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-aa-gpqa-diamond benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high |
| benchlm-aa-hle benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-lcr benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-mmmu-pro benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high |
| benchlm-aa-omniscience-accuracy benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-omniscience-hallucination-rate benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-omniscience-index benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-scicode benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-tau3-banking benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-arc-agi-1 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-arc-agi-2 benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-arc-agi-3 benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-artificial-analysis-intelligence-index benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-bug-hunt-bench benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-critpt benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-cursorbench-3-2 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-cursorbench-4-0 benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-cwe-bench-v1 benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-deepswe benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-design-arena-website benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-frontiermath-v2-tier-4 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-frontiermath-v2-tiers-1-3 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-frontierswe-v2 benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-gpqa-diamond-vals benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-livecodebench-vals benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-mazebench benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-mmlu-pro-vals benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-overall-score benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-swe-bench-pro benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high |
| benchmarklist-aa-lcr-cost-vs-score-213-models benchmarklist | — | xmodel | Grok 4.3·nothink · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·high |
| benchmarklist-arc-agi-2-of-115 benchmarklist | — | xmodel | Grok 4.6·xhigh · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·xhigh · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high |
| benchmarklist-artificial-analysis-intelligence-index-of-325 benchmarklist | — | xmodel | Grok 4.3·high · Grok 4.3·nothink |
| benchmarklist-deepswe-1-1-of-63 benchmarklist | — | xmodel | Grok 4.6·low · Grok 4.6·high · Grok 4.6·xhigh · Grok 4.6·medium |
| benchmarklist-frontiercode-2-0-of-26 benchmarklist | — | xmodel | Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high |
| benchmarklist-gpqa-diamond-of-481 benchmarklist | — | xmodel | Grok 4.3·high · Grok 4.3·nothink · Grok 4.3·low · Grok 4.3·medium |
| benchmarklist-itsmbench-of-5 benchmarklist | — | xmodel | Grok 4.5·low · Grok 4.5·medium |
| benchmarklist-obviousbench-of-275 benchmarklist | — | xmodel | Grok 4.3·high · Grok 4.3·nothink · Grok 4.3·low · Grok 4.3·medium · Grok 4.5·high · Grok 4.5·low · Grok 4.5·medium · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh |
| Brood War Bench broodwarbench | Claude Code CLI · round-robin StarCraft:Brood War tournament, 171 games, default effort only ✓ | xmodel | Grok 4.6·xhigh · Grok 4.6·medium · Grok 4.6·low |
| bsb-v1-green github-bullshitbench-v1 | — | xmodel | Grok 4.6·low · Grok 4.6·xhigh · Grok 4.5·high · Grok 4.5·low · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink |
| BVB arxiv-2609.15478 | arXiv 2609.15478 · Mini-BVB via LiteLLM, $3/scene cap, 5 Claude models at high (+ omitted-effort runs), LiteLLM list cost ✓ | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.3·high · Grok 4.3·nothink |
| c0ze/agent-evaluation: 13 tasks, blind-graded (24-way pass of 13 Oct) c0ze-agent-evaluation | 13 self-contained tasks, 1 run per model, blind rubric grading by Opus 5.5 subagents; score = mean of 13 task totals /10; cost per 13-task run (credit-balance drop, whole $; Haiku API-equivalent); read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh |
| CausalGame Agent mode (structured tool calling): mean survival rate over 14 experiments causalgame-2607.04293 | CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Agent mode (tool calls); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| CausalGame Prompting mode (Python code): mean survival rate over 14 experiments causalgame-2607.04293 | CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Prompting mode (Python code); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| Chartography (Surge AI) launch snapshot July 2026: cells absent from or different in the held board chartography-2608.10677 | Chartography launch leaderboard (arXiv 2608.10677, frozen July 2026 snapshot; 100 tasks x 20 trials, no tools, mean pass@1 % with 95% CI, Surge-hosted): only the cells that are NOT identical to the held surgehq.ai board (read 2026-10-15, group surge-chartography) are kept; join with that group for the max/xhigh peers; arXiv 2608.10677 (Aug 2026); read 2026-10-20 ✓ | xmodel | Grok 4.5·high · Grok 4.3·high |
| CL-Bench clbench.com | CL-Bench, learning from context · overall score on 0–1 with its std in the flags; Opus 4.5 with and without thinking as default/nothink; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.20·think · Grok 4.20·nothink |
| Code4Scene public set (95 cases), overall S_model arxiv-2609.36777 | Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; public set (95 cases), overall S_model; read 2026-10-19 ✓ | xmodel | Grok 4.6·high |
| CodeActionBench, Reference harness (single code-generation loop) arxiv-2609.33807 | CodeActionBench (arXiv 2609.33807, 27 Sep 2026): 25 tasks x 3 seeds = 75 attempts per configuration, agentic Code-as-Policy for embodied manipulation; 'Reference' harness for all rows of this group; score = Success (/75 attempts); task coverage /25 is another summary; cost = Table 11 total API-equivalent USD over the 75 attempts (computed by the authors from tokens at list rates); read 2026-10-20 ✓ | xmodel | Grok 4.6·high |
| Codex Radar intelligence-efficiency board, Codex CLI harness codexradar | Codex Radar recurring repository-level coding eval (per-task pass over valid tasks; payload mode equal_latest_3: equal-weight latest three valid samples per task); harness codex; snapshot payload stamped 2026-10-01T23:00:27+08:00 read 2026-10-20; score=passed/valid_tasks (derived; payload gives passed, valid_tasks, iq=150*pass_rate); cost=average_price_usd per run (estimated USD-equivalent from token prices); valid_tasks differ per row (36..336); read 2026-10-20 ✓ | xmodel | Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Grok 4.6·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low · Grok 4.7·xhigh |
| Conceptual Reasoning Index · DTBench conceptualreasoning.ai | Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.5·high · Grok 4.3·high |
| Conceptual Reasoning Index · LMCA conceptualreasoning.ai | Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.5·high · Grok 4.3·high |
| Convex evals (no guidelines, web research) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines, web research' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Grok 4.6·medium · Grok 4.5·medium |
| Convex evals (no guidelines) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Grok 4.6·medium · Grok 4.7·medium · Grok 4.5·medium |
| Convex evals (with guidelines) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'with guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Grok 4.6·medium · Grok 4.7·medium · Grok 4.5·medium |
| Convex LLM leaderboard v92a60e0b · no guidelines convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines; read 2026-10-15 ✓ | xmodel | Grok 4.6·medium · Grok 4.7·medium · Grok 4.5·medium |
| Convex LLM leaderboard v92a60e0b · no guidelines + web convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines + web; read 2026-10-15 ✓ | xmodel | Grok 4.5·medium |
| Convex LLM leaderboard v92a60e0b · with guidelines convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, with guidelines; read 2026-10-15 ✓ | xmodel | Grok 4.5·medium · Grok 4.6·medium · Grok 4.7·medium |
| ctala github-ctala | Claude Code CLI · reasoning not configured, temp 0.7 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| CTI-REALM unconstrained (300-message budget): overall reward ctirealm-2610.14180 | Unconstrained CTI-REALM (Averlon), Table 2: 50 tasks, one sample each, common harness, judges Gemma-4-31B thinking / DeepSeek-v4-flash / GLM-5.3-flash x3 trials, message budget 300 (14 models have samples run at 300 incl. 10 spliced, 12 carried from lower budgets with zero capped samples); overall weighted reward; checkpoints C0-C4 same runs (rule 8); cost = cost of the 50-task run at list prices; arXiv 2610.14180 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·high |
| CUA-SWE mobile hybrid: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, mobile domain (20 tasks), deterministic executable verifiers, access = hybrid CUA (screenshots + graphical actions); frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Grok 4.6·high |
| cwe-bench-v2-class-k01 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k02 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k03 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k04 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k05 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k06 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k07 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k08 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k09 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-class-k10 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-combined-pass1 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-combined-pass4 cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-language-c-cpp cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-language-go cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-language-java cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-language-js-ts cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-language-python cwe-bench | — | xmodel | Grok 4.7·high |
| cwe-bench-v2-language-rust cwe-bench | — | xmodel | Grok 4.7·high |
| DataLLM Lab: 9 executed Python tasks, Qwen3.8 Max family (Prime, Max, Max-0902), pass and cost per 1,000 tasks datallmlab | 9 executed Python tasks (python3 -I subprocess, 12 s timeout), temperature 0, max_tokens 4000, one scored attempt, via OpenRouter; three runs on different days (Prime 2026-10-02, Max 2026-08-20, Max-0902 2026-09-16); cost = measured tokens x list price on the priced-on date (source-computed, 'not a billing statement'); read 2026-10-20 ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.20·nothink · Grok 4.7·high |
| DeepSWE v1.1 (xAI Grok 4.6 post) xai | DeepSWE v1.1 pass@1 %, mini-swe-agent harness run by Datacurve; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| DeepSWE v1.1 (xAI Grok 4.7 post) xai | DeepSWE v1.1, resolve rate %, harness/n/attempts not stated; xAI Grok 4.7 launch post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·high · Grok 4.6·high |
| Design Arena - all categories designarena | designarena.ai/leaderboard 'allcategories': crowd human-vote Elo on AI-generated designs/UI (1,817,484 votes, 180 models); effort not in the Claude slugs (other vendors' slugs carry -max/-high) so default; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.20·think · Grok 4.20·nothink |
| Design Arena - fullstack designarena | designarena.ai/leaderboard 'fullstack' category: crowd human-vote Elo on full-stack app generation (61,252 votes, 49 models); effort not stated; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.20·think |
| Design Creativity Bench (UI generation; overall T-score, median cost per design) arxiv-2610.11539 | 80 UI design tasks / 160 prompts, one generation per design, highest effort (Opus 5.5 at xhigh); Tables 2 and 9 of arXiv 2610.11539; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| DrivingBench: progress along a 127 m real-car cone course (best attempt, %) drivingbench-2609.38948 | DrivingBench Table 2: VLM drives a real Toyota Corolla through 3 tools, one chat session per model = one trial (n=1), up to 3 attempts with retained context and a fixed reflection prompt; progress % = furthest point on the centreline (best attempt); each model in its native harness (Codex CLI 0.154.0-alpha.6.2 for GPT-6 Astra and GPT-5.6 Sol; Claude Code 2.1.274; Cursor); cost = tokens x public list prices as of Sep 2026 summed over all attempts; values typed by hand from the page-7 image (the text layer runs digits together); arXiv 2609.38948 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·medium |
| dubesor-benchtable dubesor | — | xmodel | Grok 4.20·think · Grok 4.20·nothink |
| Dyad app-builder · Deskhero github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Deskhero (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | Grok 4.6·medium · Grok 4.7·high · Grok 4.5·medium · Grok 4.6·high |
| Dyad app-builder · Portalis github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Portalis (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | Grok 4.6·medium · Grok 4.7·high · Grok 4.5·medium · Grok 4.6·high |
| Dyad app-builder · Relay CRM github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Relay CRM (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | Grok 4.6·medium · Grok 4.7·high · Grok 4.5·medium |
| E-Bench (MCP tools): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench (MCP tools): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Grok 4.5·high |
| E-Bench-Code (code execution): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench-Code (code execution): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Grok 4.5·high |
| econ-eval leaderboard A (20 tasks, Opus-5 judge) github-deluair-econ-eval | deluair/econ-eval README at commit 77f9f3e (16 Oct 2026), leaderboard of 2026-09-10: 20 economics tasks (quantitative, reasoning, coding, writing) x 5 samples = 100 completions/model, numeric/exact/code-exec graders + rubric tasks judged by Claude Opus 5; score = mean over tasks of task means; cost = total for the 100 completions from token counts x list price (prices read from the OpenRouter catalog 2026-07-31 / vendor pages); per-track tables are the same completions (not extra groups); ten models ran 2026-08-01, deepseek-flash added 2026-09-10 same tasks/samples/judge; read 2026-10-20 ✓ | xmodel | Grok 4.5·high |
| EEBench (xAI Grok 4.7 post) xai | EEBench electrical engineering accuracy %, harness/n not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high |
| EMB vals.ai | Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Epoch · chess puzzles epoch.ai | Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.7·xhigh |
| Epoch · ebr bench epoch.ai | Epoch AI's own evaluation (ebr_bench), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.6·xhigh |
| Epoch · frontiermath tier 4 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.20·think · Grok 4.5·high · Grok 4.3·high |
| Epoch · frontiermath tiers 1 3 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.20·think · Grok 4.5·high · Grok 4.3·high |
| Epoch · furniture assembly epoch.ai | Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.5·high · Grok 4.6·xhigh · Grok 4.7·xhigh |
| Epoch · gpqa diamond epoch.ai | Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.20·think · Grok 4.5·high · Grok 4.3·high |
| Epoch · mirrorcode epoch.ai | Epoch AI's own evaluation (mirrorcode), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.5·high · Grok 4.6·xhigh |
| Epoch · mystery game puzzles epoch.ai | Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| Epoch · otis mock aime 2024 2025 epoch.ai | Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.20·think · Grok 4.5·high · Grok 4.3·high |
| Epoch · simpleqa verified epoch.ai | Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·xhigh |
| epoch-ext-surface-evolver-bench epoch.ai | — | xmodel | Grok 4.5·high |
| EQ-Bench creative-writing-longform (EQ-bench/eqbench-leaderboard-results) EQ-bench/eqbench-leaderboard-results (GitHub) | GitHub repo EQ-bench/eqbench-leaderboard-results, pinned commit 266853f2ac7d10c31430cb3bd00d64c758beb581, dir creative-writing-longform/results_sonnet-4.6/, 12/12 multi-chapter longform tasks, judge claude-sonnet-4-6 (flag claude-judge), 1 iteration (n=1); distinct board from the already-held eqbench4 (eqbench.com static roleplay-Elo data, different judge/metric/source) — not a restatement.; read 2026-10-08 ✓ | xmodel | Grok 4.20·think |
| fcode20pass cognition | — | xmodel | Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh |
| fcode20score cognition | — | xmodel | Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh |
| FinancialAuditBench (2609.32835) — 45 audit workpaper tasks, bash-only agent, 8 runs/task arxiv-2609.32835 | Table 8/Table 9 main row, strict pass % and mean cost $/trajectory, 360 trajectories/model; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Grok 4.6·medium |
| FlavourBench Score (mean of 3 families x 178 tasks) flavourbench-2608.20574 | FlavourBench Table 3 leaderboard, 534 tasks, FB Score = equal-family mean, 2 collection panels pooled, OpenRouter fixed provider (GLM 5.3 direct); arXiv 2608.20574 (20 Aug 2026); family columns Table 10 are same runs (rule 8); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| FrontierCode v1.1 Extended (xAI Grok 4.6 post) xai | FrontierCode v1.1 (Extended) %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| FrontierSWE v2 frontierswe | Proximal frontierswe.com + Sonnet 5.5 card · 34 ultra-long tasks, max effort, proximus harness, mean reward@5; cost basis undocumented ✓ | xmodel | Grok 4.6·xhigh |
| GDPval (Elo) (xAI Grok 4.7 post) xai | GDPval Elo (Artificial Analysis GDPval-AA family by Elo scale); chart only; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high |
| GDPVal-AA v2 (xAI Grok 4.6 post) xai | GDPVal-AA v2 Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| gdpx-overall semianalysis | — | xmodel | Grok 4.7·xhigh |
| Gert Labs GBENCH agentic coding (models write code run as agents) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode agentic_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| Gert Labs GBENCH decision making (models play the games) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode decision_making, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| Gert Labs GBENCH one-shot coding (models write code in one shot) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode oneshot_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| Google model cards: CharXiv Reasoning (1000 val questions) | CharXiv Reasoning, 1000 reasoning questions of the validation split; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| Google model cards: GPQA Diamond (no tools) | GPQA Diamond no tools; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| Google model cards: LiveCodeBench (175 UI problems) | LiveCodeBench code generation, 175 UI problems 1 Jan-1 May 2025; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| Google model cards: Video-MMMU | Video-MMMU; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| GRIP closed-loop accuracy (E1, mean over Q1-Q5), spatial and physical reasoning for VLMs grip-2610.16170 | GRIP Table 3 Mean (E1): 34 domains, procedurally generated images with closed-form answers, 5 questions per image in fixed order, fresh single-turn per item, 8,500 queries per hosted model, routed API; hosted max output 256-2048 tokens; Q1-Q5 columns are the same runs (rule 8); arXiv 2610.16170 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| habr-crashtest27 habr-author | — | xmodel | Grok 4.7·high |
| Harvey LAB (Vals) (xAI Grok 4.6 post) xai | Harvey LAB (Vals) %, xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| Harvey Legal Agent Benchmark (xAI Grok 4.7 post) xai | Harvey Legal Agent Benchmark %, harness/judge not stated; xAI Grok 4.7 post 21 Sep 2026 (4.6 post calls it 'Harvey LAB (Vals)'); read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high |
| HealthBench Professional (xAI Grok 4.7 post) xai | HealthBench Professional % (clinical reasoning), grader/variant not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high |
| hehee9 Maze Bench Tier 1 (30 mazes) hehee9maze-bcd609c | Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 30 mazes (6 sizes x 5, 4x4..18x18), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status completed); read 2026-10-20 ✓ | xmodel | Grok 4.5·medium |
| Hy-MultiTurn (Chinese deep multi-turn dialogue understanding): Importance score hymultiturn-2607.29196 | Hy-MultiTurn leaderboard (Tencent Hunyuan authors; Hy3/Doubao rows are not in the eight providers): six dimensions, final-turn response scored against weighted rubric items; Importance (weighted correct share) = primary metric, mean of three independent rolls; arXiv 2607.29196 (31 Jul 2026); read 2026-10-20 ✓ | xmodel | Grok 4.5·high |
| hypfrontier-challenge64-hf-valid hypfrontier-2608.10843 | — | xmodel | Grok 4.5·high |
| ibmartian-full-p-grok ironbee | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| IdentityBench v3-Full: 360 multi-turn stateful IGA episodes (max 12 turns), claude_cli provider, 17 Oct 2026 github-identitybench | rlwillen0121/identity-eval-harness README fad1304 22-model v3_full table; experiments/v3-full.yaml; NEW GROUP (held identitybench-v1 is the 680-case single-prompt v1 suite); read 2026-10-18 ✓ | xmodel | Grok 4.7·high |
| IdentityBench v4.1 v41_lite github-identitybench | IdentityBench v4.1 v41_lite: 60 enterprise IGA scenarios x 6 governance categories = 360 category tasks, 1,628 required actions, 12-turn cap (T0 single monolith), seed 20261019, 1 run per model, README of 2026-10-20 (commit 56591d0), published 2026-10-19; metric tasks solved of 360; cost = run cost of the whole suite (billed or estimated); read 2026-10-20 ✓ | xmodel | Grok 4.7·high |
| IH-Benchmark instruction-hierarchy compliance, conflict scenarios (overall %) ih-benchmark-2607.25987 | IH-Benchmark Table 16 Overall: 2,336 conflict scenarios (S>U and U>T tracks), programmatic checks + 189 LLM-judge scenarios (GPT-5-mini judge), once per model, LiteLLM, max_completion_tokens 8192, provider-default sampling, Feb-May 2026; HiddenLayer; arXiv 2607.25987 (27 Jul 2026); Tables 15/17 and track/domain columns = same runs (rule 8); cost = B.3 representative cost per full-benchmark evaluation; read 2026-10-20 ✓ | xmodel | Grok 4.20·think · Grok 4.20·nothink |
| ITBench-AA (SRE) artificialanalysis | Artificial Analysis ITBench-AA (SRE); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Grok 4.3·high · Grok 4.7·xhigh · Grok 4.1 Fast·nothink |
| KernelBench CUDA 01 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 01, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| KernelBench CUDA 02 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 02, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Grok 4.7·xhigh |
| KernelBench CUDA 03 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 03, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| KernelBench CUDA 04 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 04, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Grok 4.7·xhigh |
| KernelBench hard 01_fp8_gemm github-kernelbench | kernelbench.com hard board problem 01_fp8_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh |
| KernelBench hard 03_paged_attention github-kernelbench | kernelbench.com hard board problem 03_paged_attention; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh |
| KernelBench hard 05_topk_bitonic github-kernelbench | kernelbench.com hard board problem 05_topk_bitonic; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh |
| KernelBench hard 07_w4a16_gemm github-kernelbench | kernelbench.com hard board problem 07_w4a16_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh |
| kingy930869-4tasks kingy.ai | — | xmodel | Grok 4.6·low · Grok 4.7·low |
| lb2-browser-analysis-driven-action livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-browser-constraint-adherence livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-browser-data-aggregation livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-browser-sequential-planning livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-browser-structural-integrity livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-coding-agent-loop livebench | — | xmodel | Grok 4.6·xhigh |
| lb2-coding-billing-limits livebench | — | xmodel | Grok 4.6·xhigh |
| lb2-coding-hard-fixes livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-coding-llm-api-integration livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-coding-product-ux livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-dataanalysis-consecutive-events livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-dataanalysis-table-join livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-dataanalysis-table-reformat livebench | — | xmodel | Grok 4.6·xhigh |
| lb2-design-documents-docx livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-design-pdf livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-design-presentations-ppt livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-longhorizon-delegated-operations livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-longhorizon-finance-compliance-apps livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-longhorizon-long-run-operations livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-longhorizon-operations-commerce-apps livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-longhorizon-performance-optimisation livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-longhorizon-systems-engineering livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-reasoning-logic-with-navigation livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-reasoning-spatial livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-reasoning-theory-of-mind livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-reasoning-zebra-puzzles livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-research-legal livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-research-regulatory-compliance livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-research-science-evidence livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-research-strategy livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-research-tax-finance livebench | — | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| lechmazur Adversarial LLM Debate Benchmark — Bradley-Terry rating lechmazur-debate | lechmazur/debate, side-swapped adversarial debates, multi-model judge panel (incl. a Claude judge), Bradley-Terry rating with match counts; read 2026-10-05 ✓ | xmodel | Grok 4.7·high · Grok 4.6·high · Grok 4.5·high · Grok 4.20·think · Grok 4.20·nothink |
| lechmazur Creative Writing Benchmark — comparison score lechmazur-writing | lechmazur/writing, matched creative-brief stories judged pairwise by rotating LLM evaluators, Thurstone/BT-style comparison score centered at 0, bootstrap 95% CI; read 2026-10-05 ✓ | xmodel | Grok 4.7·high · Grok 4.6·high · Grok 4.5·high |
| Lechmazur Elimination Game (TrueSkill mu) lechmazur-elimination-game | multi-player elimination tournament (public/private chat, votes, jury), TrueSkill mu aggregated over random passes; games per model printed; README leaderboard of 61 rows; changelog 'Jan 6, 2026: GPT-5.2, Opus 4.5, Gemini 3 Pro, Gemini 3 Flash, Kimi K2 Thinking, Qwen 3 Max Thinking, Grok 4.1 Fast ... added' (stale board); no cost; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| lechmazur NYT Connections Extended Benchmark — full 940-puzzle board lechmazur-nyt | lechmazur/nyt-connections, 940 puzzles, quadratic-v1 scoring (g/4)^2 per puzzle, objective (not model-judged); read 2026-10-05 ✓ | xmodel | Grok 4.20·think · Grok 4.6·xhigh · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high · Grok 4.1 Fast·nothink · Grok 4.20·nothink |
| lechmazur NYT Connections Extended Benchmark — harder 100-puzzle subset lechmazur-nyt | lechmazur/nyt-connections, harder 100-puzzle subset, same quadratic-v1 scoring, objective; read 2026-10-05 ✓ | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.7·high · Grok 4.20·nothink · Grok 4.1 Fast·nothink |
| legalforecast-mtd-micro-brier johnhughes3/LegalForecastBench | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| LiveBench 2.0 · Browser Use suite livebench | LiveBench 2.0 Browser Use suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| LiveBench 2.0 · Coding suite livebench | LiveBench 2.0 Coding suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| LiveBench 2.0 · Data Analysis suite livebench | LiveBench 2.0 Data Analysis suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| LiveBench 2.0 · Design suite livebench | LiveBench 2.0 Design suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| LiveBench 2.0 · Long-Horizon suite livebench | LiveBench 2.0 Long-Horizon suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| LiveBench 2.0 · Reasoning suite livebench | LiveBench 2.0 Reasoning suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| LiveBench 2.0 · Research suite livebench | LiveBench 2.0 Research suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.7·xhigh |
| livebench-task-amps_hard livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-code_completion livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-code_generation livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-connections livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink |
| livebench-task-consecutive_events livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-integrals_with_game livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-javascript livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-logic_with_navigation livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-math_comp livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-olympiad livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-paraphrase livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-plot_unscrambling livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-python livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-simplify livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-spatial livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-story_generation livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-summarize livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-tablejoin livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-tablereformat livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think |
| livebench-task-theory_of_mind livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-typescript livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-typos livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| livebench-task-zebra_puzzle livebench | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think |
| llm_chess — aggregated Elo vs Random + Dragon engine (all Claude models) llm_chess | maxim-saplin/llm_chess, LLM plays Black vs RANDOM_PLAYER and Dragon engine, Elo anchored to Dragon ladder, 95% CI, average_game_cost as cost per game; read 2026-10-05 ✓ | xmodel | Grok 4.6·high · Grok 4.3·high · Grok 4.20·nothink · Grok 4.1 Fast·nothink |
| llm_chess — single 10-game match vs RANDOM_PLAYER (Black win rate) llm_chess | maxim-saplin/llm_chess raw per-match _aggregate_results.json, n=10 games per model, win% computed by us from black_wins/total_games; read 2026-10-05 ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.6·high |
| llm-regression product suite 2026-10-03 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-03, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Grok 4.7·high |
| llm-regression product suite 2026-10-04 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-04, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Grok 4.7·high |
| llm-regression product suite 2026-10-05 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-05, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Grok 4.7·high |
| llm-regression product suite 2026-10-06 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-06, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Grok 4.7·high |
| llm-regression product suite 2026-10-07 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-07, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Grok 4.7·high |
| llm-stats.com arena: agents category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), agents category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Grok 4.20·think |
| llm-stats.com arena: general category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), general category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Grok 4.20·think · Grok 4.20·nothink |
| llm-stats.com arena: reasoning category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), reasoning category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Grok 4.20·think |
| llmreg-product-pass-2026-10-03 nikoamoretti-llm-regression | — | xmodel | Grok 4.7·high |
| llmreg-product-pass-2026-10-04 nikoamoretti-llm-regression | — | xmodel | Grok 4.7·high |
| llmreg-product-pass-2026-10-05 nikoamoretti-llm-regression | — | xmodel | Grok 4.7·high |
| llmreg-product-pass-2026-10-06 nikoamoretti-llm-regression | — | xmodel | Grok 4.7·high |
| llmreg-product-pass-2026-10-07 nikoamoretti-llm-regression | — | xmodel | Grok 4.7·high |
| LMArena Agent lmarena.ai | Agent Mode real sessions (Code/Chat/Work) · cost=median $/task (14d window), score=Net Improvement composite signal, effort read from model-variant label; live board, read 27 Sep 2026 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| LMArena Document lmarena.ai | Document arena (Overall, the page's default board: no style control) · blind pairwise votes on answers about uploaded documents, Elo rating, no cost; vote cutoff 13 Sep 2026, so no 5.5 model yet; effort read from the arena's model key (-thinking, -high, -max); live board, read 30 Sep 2026 ✓ | xmodel | Grok 4.5·high · Grok 4.20·think · Grok 4.6·high |
| LMArena Search lmarena.ai | Search arena (Overall, the page's default board: no style control) · blind pairwise votes on answers with web search, Elo rating, no cost; each model runs with the arena's search tool (key suffix -search); vote cutoff 24 Aug 2026, no 5.5 model; effort not in the key (default), Fable 5 at high per its display name; live board, read 30 Sep 2026 ✓ | xmodel | Grok 4.5·high |
| LMArena Text lmarena.ai | Text arena (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-opus-5.5-high-vertex); live board, read 30 Sep 2026 ✓ | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| LMArena Vision lmarena.ai | Vision arena (Overall, the page's default board: style control) · blind pairwise votes on image-input prompts, Elo rating, no cost; vote cutoff 28 Sep 2026, no 5.5 model yet; effort read from the arena's model key (-thinking, -thinking-16k/32k, -high, -max); live board, read 30 Sep 2026 ✓ | xmodel | Grok 4.5·high · Grok 4.20·think · Grok 4.6·high · Grok 4.7·xhigh |
| LMArena WebDev lmarena.ai | Code arena, WebDev board (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-sonnet-5-5-high); live board, read 30 Sep 2026 ✓ | xmodel | Grok 4.5·high · Grok 4.1 Fast·think · Grok 4.7·xhigh · Grok 4.6·high |
| LMArena WebDev — HTML sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-html, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · Grok 4.5·high · Grok 4.1 Fast·think |
| LMArena WebDev — React sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-react, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · Grok 4.5·high · Grok 4.1 Fast·think |
| M-GATE grammar error detection, MCC mean over 30 languages (temperature 0) mgate-2608.03803 | M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.1; 53 models / 82 configurations March-July 2026; linguist-crafted adversarial grammar items, 30 languages, temperature 0, one run; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓ | xmodel | Grok 4.20·think · Grok 4.3·nothink |
| M-GATE round-trip translation, hard subset, normalised score mean over 29 targets mgate-2608.03803 | M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.2; 53 models / 82 configurations March-July 2026; 100 English sources round-tripped into 29 languages, hard subset of 50 sentences, judges GPT-5.4 Mini + Claude Haiku 4.5 + Gemini 3.1 Flash Lite at temp 0; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓ | xmodel | Grok 4.20·think · Grok 4.3·nothink |
| MA-Bench mission analysis, benchmark condition, weighted score (GPT-5.6 judge), Opus 5 / Fable 5.1 / Sonnet 5 arxiv-2610.19138 | arXiv 2610.19138 Table 4; production judge GPT-5.6; 3000-word cap; 216 briefs (n=30/50/30 per model); read 2026-10-18 ✓ | xmodel | Grok 4.6·high |
| MathArena · august matharena.ai | MathArena, arxiv--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Grok 4.7·xhigh |
| MathArena · august matharena.ai | MathArena, arxiv_false--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Grok 4.7·xhigh |
| MathArena · june matharena.ai | MathArena, arxiv--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.5·high |
| MathArena · june matharena.ai | MathArena, arxiv_false--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.5·high |
| MathArena · may matharena.ai | MathArena, arxiv--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Grok 4.7·xhigh |
| MathArena · may matharena.ai | MathArena, arxiv_false--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Grok 4.7·xhigh |
| matharena-arxiv-false-september matharena.ai | — | xmodel | Grok 4.7·xhigh |
| matharena-arxiv-september matharena.ai | — | xmodel | Grok 4.7·xhigh |
| MazeBench ASCII no tools mazebench | MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, no Python tools; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| MazeBench ASCII with Python tools mazebench | MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, Python tools enabled; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh |
| Mercor · CVDP mercor.com | Mercor, CVDP: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.6·xhigh |
| Mercor · GDPval mercor.com | Mercor, GDPval: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.6·xhigh |
| Mercor · Terminal-Bench 2.1 Extended mercor.com | Mercor, Terminal-Bench 2.1 Extended: the public benchmark extended with new expert-built Mercor tasks (99 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.6·xhigh |
| Mercor APEX-Accounting mercor.com | Mercor's own board, APEX-Accounting, harness loop_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.7·xhigh |
| Mercor APEX-Agents mercor.com | Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.7·xhigh |
| Mercor APEX-SWE (inspect) mercor.com | Mercor's own board, APEX-SWE, harness inspect · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.5·high |
| Mercor APEX-v1 (The AI Productivity Index) mercor.com | Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.7·xhigh |
| mercor-apexacct-mean mercor.com | — | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.7·xhigh |
| mercor-apexv1-mean mercor.com | — | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.7·xhigh |
| Metacognition study: GPQA Diamond accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: GPQA Diamond (N=198), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: HLE accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: HLE (N=2,500), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: LiveBench accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LiveBench (N=400), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: LiveCodeBench v6 accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LiveCodeBench v6 (N=1,055), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: LiveMathBench accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LiveMathBench (N=240), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: LongBench v2 accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LongBench v2 (N=503), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: MATH-500 accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MATH-500 (N=500), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: MathVision accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MathVision (N=3,040), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: MMLU-Pro accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MMLU-Pro (N=12,032), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: MMMU-Pro accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MMMU-Pro (N=3,460), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: MuSR accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MuSR (N=756), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: OlympiadBench accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: OlympiadBench (N=6,728), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: SimpleQA accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: SimpleQA (N=4,326), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Metacognition study: ZebraLogic accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: ZebraLogic (N=1,000), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| Mine Odyssey: Minecraft spatial navigation (success rate, 180 tasks) arxiv-2610.11328 | 180 navigation tasks (125 outdoor/55 indoor), 500 decisions or 6 h; Opus 5 at 4 efforts plus Opus 5.5 max; Table 1 of arXiv 2610.11328; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh |
| MineBench voxel arena minebench | minebench.ai live API · voxel builds, 15 prompts, Elo arena read 29 Sep, measured cost for 8 of 12 Claude models ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| neocognition-apprentice-bench-api neocognition | — | xmodel | Grok 4.6·xhigh · Grok 4.5·high |
| NonoBench nonobench.com | NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓ | xmodel | Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh · Grok 4.1 Fast·think |
| nurb-bit-block github-nurb | — | xmodel | Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.6·high |
| nurb-bundle-holder github-nurb | — | xmodel | Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh |
| nurb-cable-clip github-nurb | — | xmodel | Grok 4.6·high · Grok 4.6·xhigh |
| nurb-leg-cup github-nurb | — | xmodel | Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh |
| nurb-pole-rest github-nurb | — | xmodel | Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh |
| nurb-valve-knob github-nurb | — | xmodel | Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh |
| NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections) matsonj/eval-connections | results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓ | xmodel | Grok 4.6·low · Grok 4.7·low · Grok 4.5·low |
| Parallel Search Capability · with search parallel.ai | Parallel Search Capability Leaderboard, Search Intelligence Score = mean of DSQA, HLE and WISER (100 questions each) with Parallel Search; cost per task incl. estimated search/extract usage; Claude 5.x at max effort per the page notes (Haiku 5.5 effort unstated); read 2026-10-15 ✓ | xmodel | Grok 4.7·medium |
| Parallel Search Capability · without search parallel.ai | Same suites without the search tool (model alone); inference cost only; read 2026-10-15 ✓ | xmodel | Grok 4.7·medium |
| pd-control arxiv-2608.02670 | — | xmodel | Grok 4.5·high |
| pd-nisthigh arxiv-2608.02670 | — | xmodel | Grok 4.5·high |
| pd-nonroot arxiv-2608.02670 | — | xmodel | Grok 4.5·high |
| pokebench.tv: Pokemon Red speedrun to Brock, turns/time/cost pokebench | fresh-save Pokemon Red, screenshot+RAM state per turn, budget 1000 turns, Anthropic API list-price cost; https://pokebench.tv/runs.json, src/pokebench_runs.json; read 2026-10-05 ✓ | xmodel | Grok 4.6·high · Grok 4.7·high · Grok 4.20·think · Grok 4.3·high |
| PostTrainBench v1.2 (weighted avg, 6 benchmarks x 4 base models) posttrainbench | PostTrainBench leaderboard results v1.2 (posttrainbench.com/?version=v1.2, changelog 30 Sep / 6 Oct 2026): CLI agent post-trains Qwen3-1.7B, Qwen3-4B, SmolLM3-3B, Gemma-3-4B for 10 h on one H100; score = weighted mean of AIME25, Arena-Hard writing, GPQA Main, GSM8K, HealthBench, HumanEval (BFCL dropped), final evals averaged over up to 5 seeds, contamination judges majority of 3; mean over n runs (2-3); same-benchmark-as:posttrain (paper v1 snapshot); read 2026-10-20 ✓ | xmodel | Grok 4.5·high |
| ProgramDistill-300 arxiv-2609.18805 | arXiv 2609.18805 · 300 web-app repair tasks, R2E-Gym harness, Opus 5 vs Sonnet 5 at high, tokens × LiteLLM list ✓ | xmodel | Grok 4.6·high |
| ProofBench v1.0-era values (vals.ai, Epoch relay) · accuracy % epoch.ai (vals.ai ProofBench v1.0-era) | ProofBench v1.0-era values relayed by Epoch AI (proofbench_external.csv, rows annotated 'v1.0-era value; model absent from the v1.1 board ... Not comparable to v1.1 rows'); vals.ai ProofBench v1.0 methodology (the held group valsproof is the v1.1 re-grade of 2026-08-14); cost per test as relayed; read 2026-10-20 ✓ | xmodel | Grok 4.20·think · Grok 4.3·high |
| RealityBench acceptability (1,378 frames) arxiv-2610.14397 | zero-shot VLM acceptability F1 against the rule scorer; cost = whole set at developer list price, thinking at output rate; read 2026-10-15 ✓ | xmodel | Grok 4.6·high |
| ReLE 专业能力 (professional ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/专业能力.md (专业能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| ReLE 医疗与心理健康 (medical/mental health) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/医疗与心理健康.md (医疗与心理健康); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.6·high · Grok 4.7·high · Grok 4.5·high |
| ReLE 总分 (total) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/total.md (alldata); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.7·high · Grok 4.5·high · Grok 4.6·high |
| ReLE 推理与数学计算 (reasoning/math) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/推理与数学计算.md (推理与数学计算); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| ReLE 教育 (education) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/教育.md (教育); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| ReLE 法律与行政公务 (law/admin) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/法律与行政公务.md (法律与行政公务); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| ReLE 语言与指令遵从 (language/IF) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/语言与指令遵从.md (语言与指令遵从); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.7·high · Grok 4.6·high · Grok 4.5·high |
| ReLE 通用能力 (general ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/通用能力.md (通用能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.7·high · Grok 4.5·high · Grok 4.6·high |
| ReLE 金融 (finance) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/金融.md (金融); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| ReplaySCM Block-order (100 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Block-order (100 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| ReplaySCM Hidden-order Hid-Full (250 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Hidden-order Hid-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| ReplaySCM Ord-Full (250 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Ord-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| roblox-leetcode-easy hf-roblox-llm-leaderboard | — | xmodel | Grok 4.5·medium · Grok 4.6·medium |
| roblox-leetcode-hard hf-roblox-llm-leaderboard | — | xmodel | Grok 4.5·medium · Grok 4.6·medium |
| roblox-leetcode-medium hf-roblox-llm-leaderboard | — | xmodel | Grok 4.5·medium · Grok 4.6·medium |
| roblox-qa-freeform hf-roblox-llm-leaderboard | — | xmodel | Grok 4.5·medium · Grok 4.6·medium |
| roblox-qa-multiple-choice hf-roblox-llm-leaderboard | — | xmodel | Grok 4.5·medium · Grok 4.6·medium |
| Roboflow Playground Vision Evals, OCR (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: OCR; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-ocr) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Grok 4.7·low · Grok 4.6·low · Grok 4.5·low |
| Roboflow Playground Vision Evals, overall (mean of tasks) with cost per sample roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; Overall = page's mean over the tasks scored (4/5 where OCR is missing); cost = 'Avg cost / sample' in USD from price x tokens at the pricing date 2026-10-19 ('~' = estimated from self-host/OpenRouter pricing); read 2026-10-20 ✓ | xmodel | Grok 4.7·low · Grok 4.6·low · Grok 4.5·low |
| Rubric transfer: expert score, ordinary rubric arxiv-2610.10809 | Rubric-transfer study on HealthBench-style typical split (n=105, 15 per theme over 7 themes): responses from a panel of response models are scored by judge Gemini 3.8 Flash (reasoning effort medium, majority of 3) against the expert rubric ('ordinary') or a foreign rubric; expert score 0..1 with 95% paired bootstrap CI; sampling temperature = each provider's default; ordinary expert rubric; read 2026-10-19 ✓ | xmodel | Grok 4.20·nothink |
| rubric-expert-share-lost arxiv-2610.10809 | — | xmodel | Grok 4.20·nothink |
| SenseBench (WSD) · prompt p001 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Grok 4.5·high · Grok 4.3·medium · Grok 4.3·low · Grok 4.3·high |
| SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies): MCC on primary studies sesr-eval-2610.10633 | SESR-Eval screening of SE systematic reviews (SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies)): LLM title/abstract screening via OpenRouter + Pydantic AI, MCC pooled over primary studies (Accuracy/Precision/Recall/F1/lost evidence and the secondary-study table are other summaries of the same runs: rule 8); cost = USD per secondary study (5% sample) from the secondary-study table; Claude 3.7 Sonnet is an old model whose cost is a previously recorded estimate; arXiv 2610.10633 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.20·think |
| SkillHorizon long-horizon business-agent tasks: task success (overall, all reference-workload bands) skillhorizon-2610.14966 | SkillHorizon Table 1 TS Overall: 7 executable MCP business environments, 5 reference-workload bands 64K-1M tokens, 210 executions per model and band (1,050 per model), Harbor + Inspect, context compaction at 200K, temperature/top-p unspecified; TS = proportion of tasks satisfying all final requirements; LGC (local-goal completion) and per-band columns = same runs (rule 8); arXiv 2610.14966 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·medium |
| SOC T1a concentration estimation (MAE g C/kg, lower better) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| SOC T1b concentration estimation with extra predictors (MAE g C/kg, lower better) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| SOC T2 stock calculation accuracy (%) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| SOC T3 pairwise paddock choice accuracy (%) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Grok 4.6·low |
| SpireBench: Slay the Spire Ascension 20 Ironclad, mean final floor reached arxiv-2610.12596 | 10 runs per model on the same held-out seeds, themed reskin with note-taking; Table 1 and Table 26 of arXiv 2610.12596; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh |
| stet-board24-oct stet.sh | — | xmodel | Grok 4.7·high |
| Supabase evals (with skills) supabase.com | Supabase evals in Claude Code, with skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓ | xmodel | Grok 4.7·high |
| Supabase evals (without skills) supabase.com | Supabase evals in Claude Code, without skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓ | xmodel | Grok 4.7·high |
| SuperCLUE 2025年11月 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| SuperCLUE 2026年3月 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.20·think |
| SuperCLUE 2026年3月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.20·think |
| SuperCLUE 2026年3月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.20·think |
| SuperCLUE 2026年3月 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.20·think |
| SuperCLUE 2026年3月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.20·think |
| SuperCLUE 2026年3月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Grok 4.20·think |
| Surface Evolver bench yhenon-surface-evolver | yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓ | xmodel | Grok 4.5·high |
| surfevolver-pass yhenon-surface-evolver | — | xmodel | Grok 4.5·high |
| Surge AI · Antidote: Everyday Edition surgehq.ai | Surge AI Antidote: Everyday Edition leaderboard (surgehq.ai/benchmarks/antidote); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| Surge AI · Chartography surgehq.ai | Surge AI Chartography leaderboard (surgehq.ai/benchmarks/chartography); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.7·xhigh · Grok 4.3·high |
| Surge AI · ComplexConstraints surgehq.ai | Surge AI ComplexConstraints leaderboard (surgehq.ai/benchmarks/complex-constraints); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.5·high · Grok 4.20·think |
| Surge AI · DAYJOB: Finance surgehq.ai | Surge AI DAYJOB: Finance leaderboard (surgehq.ai/benchmarks/dayjob-finance); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| Surge AI · DAYJOB: Healthcare surgehq.ai | Surge AI DAYJOB: Healthcare leaderboard (surgehq.ai/benchmarks/dayjob-healthcare); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| Surge AI · EnterpriseBench: CoreCraft surgehq.ai | Surge AI EnterpriseBench: CoreCraft leaderboard (surgehq.ai/benchmarks/enterprisebench-corecraft); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh |
| Surge AI · GDP.pdf surgehq.ai | Surge AI GDP.pdf leaderboard (surgehq.ai/benchmarks/gdp-pdf); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·xhigh · Grok 4.5·high · Grok 4.3·high |
| Surge AI · GDP.xlsx surgehq.ai | Surge AI GDP.xlsx leaderboard (surgehq.ai/benchmarks/gdp-xlsx); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh |
| Surge AI · HANDBOOK.md Agents surgehq.ai | Surge AI HANDBOOK.md Agents leaderboard (surgehq.ai/benchmarks/handbook); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.6·xhigh · Grok 4.5·high · Grok 4.3·high · Grok 4.7·xhigh |
| Surge AI · Hemingway-bench surgehq.ai | Surge AI Hemingway-bench leaderboard (surgehq.ai/benchmarks/hemingway-bench); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| Surge AI · Riemann-bench surgehq.ai | Surge AI Riemann-bench leaderboard (surgehq.ai/benchmarks/riemann-bench); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.5·high |
| Surge AI · sudo L7 surgehq.ai | Surge AI sudo L7 leaderboard (surgehq.ai/benchmarks/sudo-l7); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh |
| swe-rebench 07/26 swe-rebench | ReAct minimal · Opus5/Fable5/Sonnet5 all at high ✓ | xmodel | Grok 4.5·high |
| Terminal-Bench 4.0 tbench | tbench.ai primary payload · 66 tasks × 330 trials, Claude Code, all at max; cost basis undocumented ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · Grok 4.5·high |
| Terminal-Bench 4.0 (xAI Grok 4.7 post) xai | Terminal-Bench 4.0 'multi-hour terminal work' %, harness not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high |
| Terminal-Bench v3.0 (xAI Grok 4.6 post) xai | Terminal-Bench v3.0 %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high |
| Thinkingbox-bench pass@1 (507 tasks x20 trials, 5 stateful business domains) arxiv-2608.19741 | Thinkingbox-bench 507 tasks, N=20 trials, own MCP sandbox, simulated user GPT-5.4-mini, Azure-hosted API (local vLLM for Qwen), published 2026-08-19; cost = campaign total USD over 507x20 trials; read 2026-10-20 ✓ | xmodel | Grok 4.3·medium |
| Tinybird SQL benchmark llm-benchmark.tinybird.live | Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓ | xmodel | Grok 4.6·high · Grok 4.7·high · Grok 4.5·high |
| TLA-Verus tlaverus_algo 40 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 12: tlaverus_algo 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.5·high · Grok 4.3·high |
| TLA-Verus tlaverus_llm 40 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 13: tlaverus_llm 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.5·high · Grok 4.3·high |
| TLA-Verus verusbench 150 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 11: verusbench 150 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.5·high · Grok 4.3·high |
| Towards AI internal writing-Elo board, Sep 22, 2026 reddit-1wr03jc-img1 | 10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; top-16 leaderboard bar chart, footer 'Board of Sep 22, 2026'; values are printed bar labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓ | xmodel | Grok 4.7·high |
| TSNBench MCQA accuracy (Time-Sensitive Networking, 939 questions) tsnbench-2605.09481 | TSNBench Table 4: 939 MCQA questions on Time-Sensitive Networking, temperature 0.0 (dagger models: default settings), 3 runs for consistency; accuracy (consistency/latency/ECE/Brier/CW rate = same runs, rule 8); arXiv 2605.09481 (May 2026); read 2026-10-20 ✓ | xmodel | Grok 4.1 Fast·nothink |
| UGI Leaderboard — UGI composite (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, composite UGI score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| UGI Leaderboard — World Model factual recall (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, recipe/geoguessr/weight/music percent-error battery; test dates vary by model; read 2026-10-05 ✓ | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| UGI Leaderboard — Writing quality (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, LLM-judged Writing score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| Vals AIME vals.ai | Vals AI · AIME, compute_effort null (effort unverified), board 2026-04-16 ✓ | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·think |
| Vals CaseLaw v2 vals.ai | Vals AI · CaseLaw v2, compute_effort null (effort unverified), board 2026-05-04 ✓ | xmodel | Grok 4.1 Fast·think · Grok 4.20·think · Grok 4.3·high |
| Vals Code Migration vals.ai | Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals CorpFin v2 vals.ai | Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| Vals CyberBench vals.ai | Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals Finance Agent v2 vals.ai | Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals GPQA vals.ai | Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| Vals HLAB vals.ai | Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals Index (29 Sep) vals.ai | vals.ai Vals Index as recomputed on 29 Sep 2026 (older models dropped), max effort, accuracy and cost per test ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals IOI vals.ai | Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals Legal Research vals.ai | Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals LegalBench vals.ai | Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals LiveCodeBench vals.ai | Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| Vals MedCode vals.ai | Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals MedQA vals.ai | Vals AI · MedQA, Sonnet 4.6 vs Haiku 4.5 thinking, effort null, board 2026-04-16 ✓ | xmodel | Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·think |
| Vals MedScribe vals.ai | Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals MMLU-Pro vals.ai | Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| Vals MMMU vals.ai | Vals AI · MMMU, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high |
| Vals MortgageTax vals.ai | Vals AI · MortgageTax, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| Vals MysteryMechanism vals.ai | Vals AI · MysteryMechanism, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.6·high · Grok 4.7·xhigh |
| Vals ProgramBench vals.ai | Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.3·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals ProofBench vals.ai | Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓ | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals Public Benefits v1 vals.ai | Vals AI · Public Benefits Bench v1 (board 2026-06-09), max + Haiku default ✓ | xmodel | Grok 4.3·high |
| Vals Public Benefits v1.1 vals.ai | Vals AI · Public Benefits v1.1, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.3·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals SAGE vals.ai | Vals AI · grading student math work against a rubric, all at max ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| Vals Science Index (biology data investigation, mechanism discovery, research workflows) vals.ai | vals.ai Science Index, equal-weighted BioMysteryBench / MysteryMechanism / Terminal-Bench-Science; cost per test; read 2026-10-18 ✓ | xmodel | Grok 4.6·high · Grok 4.7·xhigh |
| Vals SRE-Bench vals.ai | Vals AI · binary reverse engineering (despite the name), all at max — snapshot 27 Sep, re-priced ✓ | xmodel | Grok 4.5·high · Grok 4.7·xhigh |
| Vals SWE-bench vals.ai | Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| Vals TaxEval v2 vals.ai | Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high |
| Vals Time Horizon KSP vals.ai | Vals AI · Time Horizon Index: KSP, 3 Claude at max, board 2026-09-14 ✓ | xmodel | Grok 4.5·high · Grok 4.6·high |
| vals.ai Robo-bench (robot arm + humanoid control in a physics sim, most tasks held out) vals.ai | claude-code harness; overall score; cost = vals 'Est. API cost' per test (USD, estimate); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh |
| vals.ai SWE-bench Multimodal (JS front-end issues with screenshots) vals.ai | Vals-run agent; resolve rate overall, cost per test (USD); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓ | xmodel | Grok 4.7·xhigh |
| valscodemig11 vals.ai | — | xmodel | Grok 4.7·xhigh |
| valsemb11 vals.ai | — | xmodel | Grok 4.7·xhigh |
| valsfab21 vals.ai | — | xmodel | Grok 4.7·xhigh |
| valsfinindex vals.ai | — | xmodel | Grok 4.7·xhigh |
| valshlab-2026-10-21 vals.ai | — | xmodel | Grok 4.7·xhigh |
| valsindex4 vals.ai | — | xmodel | Grok 4.7·xhigh |
| valslegalindex vals.ai | — | xmodel | Grok 4.7·xhigh |
| valssage11 vals.ai | — | xmodel | Grok 4.7·xhigh |
| valstaxwp vals.ai | — | xmodel | Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh |
| valswsindex vals.ai | — | xmodel | Grok 4.5·high |
| vlm-exam counting github-vlmexam | roboflow/vlm-exam · 74 samples, LLM-judge accuracy ✓ | xmodel | Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high |
| vlm-exam detection github-vlmexam | roboflow/vlm-exam · 250 images, mAP@50 ✓ | xmodel | Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high |
| vlm-exam extraction github-vlmexam | roboflow/vlm-exam · 97 samples, LLM-judge accuracy; same models as OCR ✓ | xmodel | Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high |
| vlm-exam identification github-vlmexam | roboflow/vlm-exam · 32 samples, near ceiling ✓ | xmodel | Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high |
| vlm-exam OCR github-vlmexam | roboflow/vlm-exam · 37 samples, similarity; Fable 5.1 & Opus 5.5 low/high ×3 runs, others low ×1 ✓ | xmodel | Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high |
| vlmexam-text github-vlmexam | — | xmodel | Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high |
| vulcanbench3 github-vulcanbench | — | xmodel | Grok 4.5·medium · Grok 4.5·high · Grok 4.5·low |
| WeirdML v3 (agentic, token-axis area score) epoch-archive | 11 ML tasks x 15 configurations; score = token-axis area 0-1 with 95% CI; cost = mean API $/run; agent claude_code; read 2026-10-01 ✓ | xmodel | Grok 4.7·xhigh |
| youkiti-prisma-tsuge10-accuracy github-youkiti | — | xmodel | Grok 4.6·low |
How the numbers are built — the method
Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.
In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.
The procedure
- Every number traced to its source. Each source is recorded as facts: the value as the source writes it, the exact text or table cell it is read from, the URL and the date of the capture; agents read every value line by line, and a measurement whose facts lack their proof stays out. Registries of models (the names each was served under, its effort settings and defaults, dated), routes (makers' APIs, resellers, routers, harnesses), benchmarks and their versions, publishers and dated prices let one build decide, by a fixed decision tree, which model, configuration and price each measurement belongs to.
- Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale, one version of a board; a group that mixes them is split. Every run is dated, and its model and effort are read as the maker and every route in between served them on that date — never assumed: a route the source does not name is never presumed to be the maker's API, and every route that could have served the run must give the same configuration. At the least doubt on the model, the effort or what was run, the measurement stays in the data file and out of the fit.
- Costs at today's prices. Every cost is put on the model's current price from the prices in force on its run date: the source's own figure at that price, the price change applied to the run's own token counts, or, when the token mix is unknown, a price band whose width is added to the cost's precision. A cost that cannot be brought to the current price stays out.
- Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The scale is read from the metric's label, never from the benchmark's identity; whether lower is better is read from the benchmark's registry, and such a score is turned in its own unit first (an attack success rate of 30 % reads 70 %); a score whose direction is not established stays out. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
- One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result. The sizes of these effects are estimated for each model maker on its own, not tied to the other makers'; a maker's own charts about its own models carry an estimated home effect, also each maker's own, left out of every value. The fit takes the couples of eight makers measured in the same groups (Anthropic, OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi), on one scale; this page shows the xAI couples. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
- Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source; the one weight set by rule is that of a single run: when a source gives k quality measurements of one run (several metrics, sub-scores or slices of the same executions), each weighs 1/k, so the run counts as one measurement.
- Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, a board copying another's numbers, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. Versions of one board that re-grade the same runs (recognised by identical costs) count those runs once. Before the fit, two groups that still share identical costs on three couples or more stop it until the copy is resolved or the coincidence proven from the sources. An index keeps, for a couple whose index components are already in the data, only what its missing components add, or is left out.
- Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
- From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.
Key values
- Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
- Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
- Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit five figures on this page can move by one unit in that digit (the cost of Grok 4.6 at high, the cost of Grok 4.6 at xhigh, the cost of Grok 4.7 at high, the cost of Grok 4.7 at medium, the cost of Grok 4.7 at xhigh), far inside their intervals.
- Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.
Techniques and software
- Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes, six chains whose draws continue in the same run until the criteria are met (CmdStan only to resume a stopped fit from its stored draws, or to restart a chain left in a far region from a good chain's last draw); built and checked stage by stage following the Bayesian workflow.
- Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
- Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
- Scales: the empirical logit of bounded scores, with the binomial shape of its noise.
Checks (on the fit of 29 September 2026, before the other makers' couples joined the data). Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.