Quality vs. cost of every model, at each effort level
One point per couple: a model at one configuration (thinking off, minimal, low → max, thinking on); no line joins a model's couples. Only the couples of the Pareto frontier are labelled (model · effort), where there is room; hover or tap any point for its model, effort, cost and score. Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward. A model's builds and modes — a dated build, a preview, a Pro mode — share its colour: the first is drawn with filled points, the others with hollow points, and its label and tooltip say what it is. They are separate couples, each with its own effort ladder.
Best model for the price — the Pareto frontier
The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated; the frontier couples are labelled (model · effort) where there is room — hover or tap any point for its model, effort, cost and score. No line joins a model's couples. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.
Value — distance to the price trend
The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 83 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.71, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.
| Model · effort | cost | expected score | value index trend = 100 |
|---|---|---|---|
| GPT-6.1 Terra · xHigh | 34× | 69.6 % | 387 |
| GPT-6.1 Terra · Medium | 20× | 66.8 % | 384 |
| GPT-6.1 Terra · High | 27× | 68.3 % | 378 |
| GPT-6 Luna · Medium | 2.4× | 54.2 % | 317 |
| GPT-6.1 Terra · Low | 11× | 62.7 % | 310 |
| GPT-6 Luna · High | 3.5× | 56.1 % | 309 |
| GPT-6 Luna · Max | 8.1× | 60.8 % | 308 |
| GPT-6 Luna · xHigh | 4.8× | 57.8 % | 305 |
| GPT-6.1 Sol · High | 54× | 70.0 % | 261 |
| GPT-6.1 Sol · xHigh | 69× | 71.0 % | 254 |
| Claude Haiku 5.5 · High | 7.9× | 59.3 % | 243 |
| GPT-6.1 Sol · Max | 84× | 71.7 % | 241 |
| GPT-6 Luna · Low | 1.8× | 48.9 % | 159 |
| Claude Fable 5.2 · Medium | 191× | 73.2 % | 150 |
| Claude Fable 5.2 · Low | 165× | 71.8 % | 127 |
| Gemini 4 Carbon · Medium | 342×promo 171× | 74.8 % | 123 |
| Claude Fable 5.2 · High | 366× | 74.9 % | 119 |
| Gemini 4 Carbon · High | 439×promo 220× | 75.3 % | 110 |
| Claude Fable 5.2 · xHigh | 569× | 75.5 % | 90 |
| GPT-6 Luna · Thinking off | 1.3× | 44.4 % | 88 |
| Qwen3.8 Flash · Thinking off | 1.3× | 43.5 % | 73 |
| Claude Fable 5.2 · Max | 1251× | 76.5 % | 53 |
| Qwen3.7 Flash · Thinking off | 1.0× | 40.5 % | 46 |
Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.
Best value by task tier — what to actually run
Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.
Picked as the frontier couple that sits furthest below the price trend : it costs 3.9× less than the trend charges for its quality — or, read on the other axis, it scores 7.1 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.
Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — effort levels compared
Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.
| Model | thinking off | minimal | low | medium | high | xhigh | max | thinking on | Expected score top effort |
|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 | — | — | 165144–189 | 191167–219 | 366322–415 | 569495–654 | 12511101–1422 | — | 76.5 % |
| Gemini 4 Carbon | — | — | 391294–520promo 196× | 342258–454promo 171× | 439340–567promo 220× | — | — | — | 75.3 % |
| Claude Opus 5.5 | — | — | 8378–89 | 120114–127 | 151143–159 | 268251–287 | 676629–726 | — | 73.9 % |
| Gemini 4 Hydrogen | — | — | 138.3–20 | 6248–79 | 160130–197 | 221166–294 | — | — | 72.8 % |
| Claude Sonnet 5.5 | — | — | 4238–46 | 4642–51 | 6761–73 | 129115–145 | 428374–489 | — | 72.6 % |
| GPT-6 Astra | — | — | 160149–172 | 207193–223 | 287270–306 | 359334–387 | 458424–495 | — | 72.6 % |
| GPT-6.1 Sol | — | — | 3128–33 | 3835–41 | 5451–59 | 6963–75 | 8477–91 | — | 71.7 % |
| GPT-6.1 Terra | — | — | 118.0–16 | 2016–25 | 2721–34 | 3426–43 | 11091–133 | — | 71.0 % |
| Gemini 4 Argon | — | — | 8372–95promo 41× | 10590–124promo 53× | 292263–324promo 146× | — | — | — | 70.9 % |
| Qwen3.8 Max | — | — | 7854–111 | — | — | 251184–341 | — | — | 66.7 % |
| Grok 4.7 | — | — | 10188–116 | 199174–227 | 293262–326 | 320281–364 | — | — | 65.0 % |
| Claude Haiku 5.5 | 2.92.4–3.4 | — | 3.53.2–3.8 | 5.04.6–5.5 | 7.97.2–8.7 | 1413–16 | 4238–47 | — | 63.9 % |
| DeepSeek V4.1 Flash | — | — | 1612–23off-peak 8.2× | — | 116.9–17off-peak 5.3× | — | 2721–36off-peak 14× | — | 63.3 % |
| DeepSeek V4.1 Flash 1017 build | 4.72.7–8.1off-peak 2.3× | — | 1410–18off-peak 6.8× | — | 1612–21off-peak 8.0× | — | 2016–25off-peak 10.0× | — | 62.6 % |
| Qwen3.8 Max 0902 build | — | — | 7757–106 | 7955–111 | — | 10375–142 | — | — | 62.5 % |
| Grok 4.6 | — | — | 4641–50 | 117105–129 | 135125–146 | 149134–165 | — | — | 62.3 % |
| Kimi K3 | — | — | 6253–72 | — | 204120–349 | — | 228206–253 | — | 62.1 % |
| GPT-6 Luna | 1.31.1–1.5 | — | 1.81.6–2.0 | 2.42.2–2.7 | 3.53.2–3.9 | 4.84.4–5.4 | 8.17.3–8.9 | — | 60.8 % |
| Gemini 3.7 Flash | — | — | 2621–32 | 4133–50 | — | — | — | — | 60.7 % |
| GLM-5.3 | — | — | 3023–38 | — | 6648–89 | — | 10488–122 | — | 60.5 % |
| GLM-5.3-Flash | — | — | 3.22.4–4.1 | — | 5.03.8–6.4 | — | 9.37.2–12 | — | 58.7 % |
| Grok 4.5 | — | — | 4538–53 | 7360–89 | 7262–85 | — | — | — | 56.7 % |
| Kimi K2.7 Code | 70merged · 57–86 | 54.9 % | |||||||
| Gemini 3 Flash | — | 9.47.5–12 | 108.7–12 | — | — | — | — | — | 52.9 % |
| Gemini 3.5 Flash-Lite | — | 7.75.7–10 | 6.25.2–7.3 | 1512–18 | 2925–34 | — | — | — | 51.3 % |
| Qwen3.8-27B | — | — | 3024–38 | 3926–59 | — | 8055–118 | — | — | 50.1 % |
| Grok 4.3 | — | — | 4234–51 | — | — | — | — | — | 48.5 % |
| Gemini 3.1 Flash-Lite | — | 2.72.0–3.7 | — | 5.13.8–6.8 | — | — | — | — | 48.1 % |
| DeepSeek V4 Flash preview | — | — | — | — | 8.25.3–13 | — | — | — | 47.0 % |
| Kimi K2 Thinking | 53merged · 35–82 | 46.2 % | |||||||
| Kimi K2.6 | 2419–29 | — | — | — | — | — | — | — | 43.6 % |
| Qwen3.8 Flash | 1.30.95–1.7 | — | — | — | — | — | — | — | 43.5 % |
| Grok 4.1 Fast | 1.20.65–2.0 | — | — | — | — | — | — | 126.1–22 | 43.4 % |
| Qwen3.7 Flash | 1.00.74–1.4 | — | — | — | — | — | — | — | 40.5 % |
| Grok 4.20 | 9.15.9–14 | — | — | — | — | — | — | — | 38.2 % |
"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. A model measured at a single setting (solo : no effort parameter, see the measurement sources) fills one merged cell. A thinking off column appears when a model is measured with its thinking switched off, a minimal one for the lowest level some models have below low; each is a configuration of its own, read from its own measurements. n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).
Value-index matrix — every model and effort against the price trend
The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.
| Model | thinking off | minimal | low | medium | high | xhigh | max | thinking on |
|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 | — | — | 12798–166 | 150115–194 | 11994–150 | 9070–115 | 5342–68 | — |
| Gemini 4 Carbon | — | — | 4125–65promo 81 | 12380–187promo 246 | 11076–160promo 220 | — | — | — |
| Claude Opus 5.5 | — | — | 7262–83 | 122108–137 | 128114–143 | 10693–121 | 5044–58 | — |
| Gemini 4 Hydrogen | — | — | 12453–290 | 15191–251 | 12073–199 | 11969–205 | — | — |
| Claude Sonnet 5.5 | — | — | 4236–51 | 6354–75 | 10590–122 | 120100–143 | 5949–71 | — |
| GPT-6 Astra | — | — | 5952–67 | 7667–86 | 6255–70 | 6255–71 | 5448–62 | — |
| GPT-6.1 Sol | — | — | 193165–225 | 257222–298 | 261230–297 | 254217–297 | 241210–277 | — |
| GPT-6.1 Terra | — | — | 310196–492 | 384268–548 | 378263–542 | 387264–565 | 158117–212 | — |
| Gemini 4 Argon | — | — | 10676–148promo 212 | 9672–128promo 192 | 5948–72promo 118 | — | — | — |
| Qwen3.8 Max | — | — | 3518–65 | — | — | 3017–52 | — | — |
| Grok 4.7 | — | — | 1913–27 | 1611–24 | 1210–16 | 1713–21 | — | — |
| Claude Haiku 5.5 | 3923–64 | — | 172142–210 | 244204–291 | 243206–287 | 211174–256 | 10387–123 | — |
| DeepSeek V4.1 Flash | — | — | 8655–135off-peak 173 | — | 14085–231off-peak 281 | — | 144104–200off-peak 289 | — |
| DeepSeek V4.1 Flash 1017 build | 208–49off-peak 40 | — | 201133–303off-peak 402 | — | 165116–233off-peak 330 | — | 173128–233off-peak 346 | — |
| Qwen3.8 Max 0902 build | — | — | 2314–38 | 6425–168 | — | 3320–56 | — | — |
| Grok 4.6 | — | — | 2418–33 | 2821–36 | 1512–19 | 2217–29 | — | — |
| Kimi K3 | — | — | 85–13 | — | 116–21 | — | 1412–17 | — |
| GPT-6 Luna | 8864–122 | — | 159129–197 | 317268–374 | 309265–360 | 305260–358 | 308263–360 | — |
| Gemini 3.7 Flash | — | — | 8159–111 | 6144–85 | — | — | — | — |
| GLM-5.3 | — | — | 1711–26 | — | 2113–34 | — | 2318–29 | — |
| GLM-5.3-Flash | — | — | 8657–129 | — | 12785–189 | — | 186129–266 | — |
| Grok 4.5 | — | — | 1611–23 | 1811–28 | 1713–22 | — | — | — |
| Kimi K2.7 Code | 139–17 | |||||||
| Gemini 3 Flash | — | 4728–79 | 5940–87 | — | — | — | — | — |
| Gemini 3.5 Flash-Lite | — | 3219–54 | 1811–31 | 158–27 | 1611–22 | — | — | — |
| Qwen3.8-27B | — | — | 106–16 | 138–23 | — | 43–7 | — | — |
| Grok 4.3 | — | — | 64–11 | — | — | — | — | — |
| Gemini 3.1 Flash-Lite | — | 3317–67 | — | 4725–89 | — | — | — | — |
| DeepSeek V4 Flash preview | — | — | — | — | 2412–45 | — | — | — |
| Kimi K2 Thinking | 32–5 | |||||||
| Kimi K2.6 | 43–6 | — | — | — | — | — | — | — |
| Qwen3.8 Flash | 7331–171 | — | — | — | — | — | — | — |
| Grok 4.1 Fast | 63–15 | — | — | — | — | — | — | 84–16 |
| Qwen3.7 Flash | 4627–79 | — | — | — | — | — | — | — |
| Grok 4.20 | 31–5 | — | — | — | — | — | — | — |
Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.
What moves everything — four factors behind the numbers
The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.
Task type
Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.
Complexity
The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).
Generation
Each release buys the same quality for less: turn on Older models to set each family's earlier models beside its current one, on the same scales.
Cache & harness
Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.
Data Sources
Every source below measured at least two (model, effort) couples on the same task; 10115 cost and 23548 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.
| Source | Actual config (harness · effort) | Type | Linked couples (same task) |
|---|---|---|---|
| AA CritPt artificialanalysis | AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Claude Opus 5.5·low · DeepSeek V4.1 Flash 1017 build·nothink · Claude Opus 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · GPT-5.5 Pro·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Gemini 3.8 Flash·low · Kimi K2.6·nothink · Grok 4.1 Fast·think · GPT-5.4 mini·medium · Qwen3.6-27B·nothink · GLM-5·think · GPT-5.4·nothink · GPT-5.2·xhigh · GPT-5.1·high · Qwen3.5-27B·nothink · GPT-5.4·low · Gemini 3.1 Flash-Lite·high · GPT-5.3-Codex·xhigh · Grok 4.3·medium · GPT-5.2·nothink · GPT-5.5·nothink · GLM-4.7·think · GPT-5.6 Sol·nothink · GPT-5.2·medium · GPT-5.4 nano·medium · GPT-5.4·xhigh · GPT-5.5·low · Gemini 3.7 Flash·low · Claude Opus 4.7·max · Gemini 3.7 Flash·medium · Grok 4.3·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Grok 4.20·think · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| AA GDP-PDF artificialanalysis | AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓ | sweep | Claude Fable 5.1·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · GPT-5.6 Sol·nothink · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5·low · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·high · Claude Sonnet 5·max · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| AA GDPval v2 artificialanalysis | AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓ | sweep | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · Claude Opus 5.5·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low · GPT-6 Luna·nothink · Qwen3.8-27B·medium · Gemini 3.8 Flash·medium · GPT-6 Luna·high · Qwen3.8-27B·low · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · GPT-6 Astra·max · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Haiku 5.5·max · GPT-6.1 Terra·max · GPT-6.1 Sol·max · Gemini 4 Argon·high · Claude Fable 5.2·max · Claude Fable 5.2·xhigh · Claude Fable 5.2·high · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · GLM-5.1·think · GLM-5.3-Flash·max · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · Claude Opus 5·low · Claude Sonnet 5·low · Claude Opus 5·medium · Claude Sonnet 5·medium · Claude Opus 5·high · Claude Sonnet 5·high · Claude Opus 5·xhigh · Claude Sonnet 5·xhigh · Qwen3.6-27B·nothink · GPT-5.1·high · Gemini 3.1 Flash-Lite·high · GPT-5.5·nothink · GLM-4.7·think · GPT-5.6 Sol·nothink · GPT-5.4·xhigh · GPT-5.4 mini·nothink · GPT-5.5·low · Gemini 3.7 Flash·low · Claude Opus 4.7·max · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink |
| AA Harvey LAB artificialanalysis | AA model payload · Harvey's Legal Agent Bench (AA's own run, separate from Vals's HLAB), score-only, 8 models × ladder + 4 at max ✓ | sweep | Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5.5·max · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · Grok 4.3·high · Grok 4.5·high · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA HLE artificialanalysis | AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓ | sweep | Claude Fable 5.1·low · Claude Opus 5.5·low · DeepSeek V4.1 Flash 1017 build·nothink · Claude Opus 5.5·xhigh · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Gemini 3.8 Flash·low · Kimi K2.6·nothink · Grok 4.1 Fast·think · GPT-5.4 mini·medium · Qwen3.6-27B·nothink · GLM-5·think · GPT-5.4·nothink · GPT-5.2·xhigh · Grok 4.1 Fast·nothink · GPT-5.1·high · Qwen3.5-27B·nothink · GPT-5.4·low · Gemini 3.1 Flash-Lite·high · GPT-5.3-Codex·xhigh · Grok 4.3·medium · GPT-5.2·nothink · GPT-5.5·nothink · GLM-4.7·think · GPT-5.6 Sol·nothink · GPT-5.2·medium · GPT-5.4 nano·medium · GPT-5.4·xhigh · GPT-5.4 mini·nothink · Claude Haiku 4.5·nothink · GPT-5.5·low · Gemini 3.7 Flash·low · Claude Opus 4.7·max · Gemini 3.7 Flash·medium · Grok 4.3·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · GPT-5.4 nano·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Grok 4.20·think · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·max |
| AA Index artificialanalysis | AA-Index · cross-model at max + Opus 5 sweep low→max ✓ | sweep | DeepSeek V4 Pro preview·high · Gemini 3.5 Flash·high · GLM-4.7·think · GPT-5.1·high · GPT-5.4·xhigh · GPT-5.5·low · GPT-5.5·nothink · GPT-5.6 Sol·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink |
| AA Index v4 artificialanalysis | AA model pages · Fable 5.1 sweep low→max + Fable 5/Opus 5 at max, per-suite $ ✓ | sweep | Kimi K2 Thinking·solo · Gemini 3.5 Flash·high · Gemini 3.7 Flash·high |
| AA MLCR artificialanalysis | AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓ | sweep | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · GLM-5.3·max · GLM-5.3-Flash·max · GPT-5.4 mini·xhigh · GPT-5.5·xhigh · GPT-5.5·low · GPT-5.5·nothink · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Gemini 3.5 Flash·high · GPT-5.4 nano·xhigh |
| AA Omniscience artificialanalysis | AA model payload · Omniscience accuracy (not the penalised index) ✓ | sweep | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · Claude Opus 5.5·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3 Pro·high · GLM-4.7·think · GLM-5·think · GLM-5.1·think · GLM-5.3-Flash·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.4·low · GPT-5.4 mini·xhigh · GPT-5.4 mini·medium · GPT-5.4 mini·nothink · GPT-5.4 nano·xhigh · GPT-5.4 nano·medium · GPT-5.4 nano·nothink · GPT-5.4·nothink · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·nothink · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.20·think · Grok 4.20·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.5-122B-A10B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Gemini 3.1 Flash-Lite·high · Claude Haiku 4.5·nothink · Claude Opus 4.7·max · Claude Opus 5·high |
| AA SciCode artificialanalysis | AA model payload · SciCode, 4 models × ladder + 3 at max ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · DeepSeek V4.1 Flash 1017 build·nothink · Claude Opus 5.5·xhigh · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · GPT-5.6 Sol·nothink · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.5 Flash·high · Claude Opus 5.5·high · Claude Opus 5·low · Claude Fable 5.1·high · Claude Fable 5.1·max · Claude Fable 5·max · Claude Sonnet 5·low · Claude Sonnet 5·high |
| AA TB 4.0 artificialanalysis | AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Claude Opus 5.5·xhigh · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| AA Terminal-Bench-Science artificialanalysis | AA model payload · Terminal-Bench-Science 0.1 (AA harness, separate from Vals's board), score-only, 4 models × ladder + 2 at max ✓ | sweep | GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · GPT-6 Luna·medium · GPT-6.1 Sol·high · Grok 4.7·xhigh · GPT-6 Luna·nothink · Gemini 3.8 Flash·medium · GPT-6 Luna·high · GPT-6.1 Sol·medium · GPT-6 Astra·low · GPT-6 Luna·low · GLM-5.3·low · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · GLM-5.3·max · GLM-5.3-Flash·max · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.6·high · Grok 4.7·high · Grok 4.7·low · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · DeepSeek V4 Pro 0813 build·max · Claude Opus 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Haiku 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| AA-Briefcase artificialanalysis | AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·medium · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Claude Haiku 5.5·max · Claude Sonnet 5.5·high · Claude Sonnet 5.5·max · Claude Opus 5.5·high · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 5·low · Claude Sonnet 5·low · Claude Opus 5·medium · Claude Sonnet 5·medium · Claude Opus 5·high · Claude Sonnet 5·high · Claude Opus 5·xhigh · Claude Sonnet 5·xhigh · Gemini 3.1 Flash-Lite·high |
| AA-LCR artificialanalysis | AA model payload · long-context reasoning, input-dominated cost ✓ | sweep | Claude Fable 5.1·medium · DeepSeek V4.1 Flash 1017 build·nothink · Claude Opus 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Gemini 3.8 Flash·low · Kimi K2.6·nothink · Grok 4.1 Fast·think · GPT-5.4 mini·medium · Qwen3.6-27B·nothink · GLM-5·think · GPT-5.4·nothink · GPT-5.2·xhigh · Grok 4.1 Fast·nothink · GPT-5.1·high · Qwen3.5-27B·nothink · GPT-5.4·low · Gemini 3.1 Flash-Lite·high · GPT-5.3-Codex·xhigh · Grok 4.3·medium · GPT-5.2·nothink · GPT-5.5·nothink · GLM-4.7·think · GPT-5.6 Sol·nothink · GPT-5.2·medium · GPT-5.4 nano·medium · GPT-5.4·xhigh · GPT-5.4 mini·nothink · Claude Haiku 4.5·nothink · GPT-5.5·low · Gemini 3.7 Flash·low · Claude Opus 4.7·max · Gemini 3.7 Flash·medium · Grok 4.3·low · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · GPT-5.4 nano·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·medium · Claude Sonnet 5·high |
| Agents on Rails · features rubyonrails | rubyonrails.org/ai + rails/ai-evals runs · 20 tickets ×3, miniswen; Opus 5 & Fable 5.1 high→max, Opus 5.5 default ✓ | sweep | GPT-6 Astra·max · GPT-6 Astra·medium · GPT-5.6 Sol·max · Gemini 3.8 Flash·medium · GPT-5.6 Luna·max · Gemini 3.8 Flash·high · GPT-5.6 Sol·medium · Grok 4.6·xhigh · GLM-5.3-Flash·max · Grok 4.6·high · Kimi K3·max · GPT-5.6 Luna·medium · Claude Opus 5.5·medium · Claude Fable 5.1·high · Claude Fable 5.1·max · Claude Opus 5·high · Claude Opus 5·max |
| AI4AI-Bench arxiv-2608.20318 | arXiv 2608.20318 · 10 ML-research repos, measured exploration spend, 2 models × ladder ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max |
| Angular shopping cart, Opus 5.5 effort ladder (OctAcademy video) youtube-octacademy | Claude Code v2.1.280; same CLAUDE.md and prompt; fresh branch per run; one run per level; 10-point checklist scored by a fresh Claude session; 2026-10-16; read 2026-10-18 ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| Anthropic docs: max_tokens cap vs cost (internal repo tasks) (16k cap) anthropic | ~135 internal repository tasks ('cap measurement'); Opus 5.5 at its default (medium), Fable 5.1 at its default (high); cost = whole-dollar bar labels, digitised from cost-intel-max-tokens-saving-opus-5-5.png; read 2026-10-01 ✓; 16k max_tokens cap only; score only | sweep | Claude Opus 5.5·medium · Claude Fable 5.1·high |
| Anthropic docs: max_tokens cap vs cost (internal repo tasks) (64k cap) anthropic | ~135 internal repository tasks ('cap measurement'); Opus 5.5 at its default (medium), Fable 5.1 at its default (high); cost = whole-dollar bar labels, digitised from cost-intel-max-tokens-saving-opus-5-5.png; read 2026-10-01 ✓; 64k max_tokens cap only; score only | sweep | Claude Opus 5.5·medium · Claude Fable 5.1·high |
| ARC-AGI-1 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓ | sweep | Claude Haiku 4.5·nothink · GPT-5.2·nothink · DeepSeek V4 Pro 0813 build·low · Qwen3.8-27B·xhigh · GPT-5.1·low · GPT-5.1·medium · GPT-5.1·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.2·high · GPT-5.2·xhigh · GPT-5.2 Pro·medium · GPT-5.2 Pro·high · GPT-5.2 Pro·xhigh · Gemini 3 Flash·minimal · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high · Grok 4.20·think · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·max · Claude Sonnet 4.6·high · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 Pro·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·medium · GPT-5.4 mini·low · GPT-5.4 nano·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·medium · GPT-5.4 nano·low · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5 Pro·xhigh · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Grok 4.5·high · Grok 4.5·medium · Grok 4.5·low · Claude Opus 5·high · Claude Opus 5·max · Kimi K3·max · Claude Fable 5·max · Claude Fable 5·xhigh · Claude Fable 5·high · Claude Fable 5·medium · Claude Fable 5·low · DeepSeek V4 Flash 0731 build·high · GPT-5.6 Luna·max · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·medium · GPT-5.6 Luna·low · GPT-5.6 Luna·nothink · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.6 Flash·minimal · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Gemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · DeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·high · Claude Fable 5.1·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Fable 5.1·medium · Claude Fable 5.1·low · GPT-6 Astra·max · GPT-6 Astra·xhigh · GPT-6 Astra·high · GPT-6 Astra·medium · GPT-6 Astra·low · GPT-6 Luna·max · GPT-6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Luna·medium · GPT-6 Luna·low · GPT-6 Luna·nothink · Claude Opus 5.5·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·low · GPT-6 Sol·max · GPT-6 Sol·xhigh · GPT-6 Sol·high · GPT-6 Sol·medium · GPT-6 Sol·low · GPT-6 Sol·nothink · GLM-5.3-Flash·max · GLM-5.3-Flash·high · GLM-5.3-Flash·low · GPT-6.1 Sol·max · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·high · GPT-6.1 Sol·medium · GPT-6.1 Sol·low · Qwen3.8-27B·medium · Qwen3.8-27B·low · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·low · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low · Claude Haiku 5.5·max · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·high · Claude Haiku 5.5·medium · Claude Haiku 5.5·low · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·nothink · Claude Sonnet 5.5·max · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Fable 5.2·xhigh · Claude Fable 5.2·high · Claude Fable 5.2·medium · Claude Fable 5.2·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| ARC-AGI-2 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓ | sweep | Claude Haiku 4.5·nothink · GPT-5.2·nothink · DeepSeek V4 Pro 0813 build·low · Qwen3.8-27B·xhigh · GPT-5.1·low · GPT-5.1·medium · GPT-5.1·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.2·high · GPT-5.2·xhigh · GPT-5.2 Pro·medium · GPT-5.2 Pro·high · Gemini 3 Flash·minimal · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high · Grok 4.20·think · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Sonnet 4.6·high · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 Pro·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·medium · GPT-5.4 mini·low · GPT-5.4 nano·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·medium · GPT-5.4 nano·low · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5 Pro·xhigh · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Grok 4.5·high · Grok 4.5·medium · Grok 4.5·low · Claude Opus 5·high · Claude Opus 5·max · Kimi K3·max · Claude Fable 5·max · Claude Fable 5·xhigh · Claude Fable 5·high · Claude Fable 5·medium · Claude Fable 5·low · DeepSeek V4 Flash 0731 build·high · GPT-5.6 Luna·max · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·medium · GPT-5.6 Luna·low · GPT-5.6 Luna·nothink · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.6 Flash·minimal · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Gemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · DeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·high · Claude Fable 5.1·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Fable 5.1·medium · Claude Fable 5.1·low · GPT-6 Astra·max · GPT-6 Astra·xhigh · GPT-6 Astra·high · GPT-6 Astra·medium · GPT-6 Astra·low · GPT-6 Luna·max · GPT-6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Luna·medium · GPT-6 Luna·low · GPT-6 Luna·nothink · Claude Opus 5.5·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·low · GPT-6 Sol·max · GPT-6 Sol·xhigh · GPT-6 Sol·high · GPT-6 Sol·medium · GPT-6 Sol·low · GPT-6 Sol·nothink · GLM-5.3-Flash·max · GLM-5.3-Flash·high · GLM-5.3-Flash·low · GPT-6.1 Sol·max · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·high · GPT-6.1 Sol·medium · GPT-6.1 Sol·low · Qwen3.8-27B·medium · Qwen3.8-27B·low · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·low · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·high · Claude Haiku 5.5·medium · Claude Haiku 5.5·low · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·nothink · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Fable 5.2·xhigh · Claude Fable 5.2·high · Claude Fable 5.2·medium · Claude Fable 5.2·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| ARC-AGI-3 public set, GPT-5.6 Sol, Harness with retained reasoning and compaction openai | ARC-AGI-3 public set, RHAE score; GPT-5.6 Sol; Harness with retained reasoning and compaction; five efforts; tokens = output tokens per game; published ~2026-07-30 (first valid Wayback capture); OpenAI research post; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| ARC-AGI-3 public set, GPT-5.6 Sol, Official harness openai | ARC-AGI-3 public set, RHAE score; GPT-5.6 Sol; Official harness; five efforts; tokens = output tokens per game; published ~2026-07-30 (first valid Wayback capture); OpenAI research post; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| arXiv 2609.13463 · TRAIL root-cause attribution arxiv-2609.13463 | Root-cause attribution of agent failures on TRAIL, Continual Search judge in the Claude Agent SDK, turn 4 · weighted F1 and cumulative mean cost per trajectory (USD), read from the vector coordinates of figures 4–5 (they agree with the numbers the text states); Opus 4.8 and Sonnet 5 at low/high/max (figure 5), Fable 5 at default settings (figure 4, whose Opus and Sonnet points are figure 5's high runs); read 30 Sep 2026 ✓ | sweep | Claude Opus 4.8·low · Claude Opus 4.8·high · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·high · Claude Sonnet 5·max · GPT-5.5·xhigh |
| ArXivMath no-tools anthropic | Opus 5.5 card p182 · ArXivMath Aug 2026, 57 problems, 3 models × low→max, $ cost, scores printed ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max |
| ArXivMath tools anthropic | Opus 5.5 card p183 · ArXivMath Aug 2026 with code sandbox, 3 models × low→max, scores printed ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max |
| AutomationB. 5 anthropic | Opus 5 card p180 · Zapier, linear-$ chart; Fable=Mythos 5, effort inferred ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max |
| BBQ effort (interface-gap) arxiv-2609.08861 | direct API · budget_tokens low/med/high, Sonnet4.6 & Haiku4.5 ✓ | sweep | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high |
| BenchCAD V2C anthropic | Opus 5.5 card p205 · Vision2Code with tools, voxel IoU, 4 models × 5 efforts; effort inferred from cost order ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| BenchCAD V2C −tools anthropic | Sonnet 5.5 card p129 · tools disabled, 4 models × 5, separate regime kept out of the effort grid ✓ | sweep | Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max |
| BrowseComp 10M anthropic | Opus 5 card p160 · 10M budget, sweep low→max; Fable=Mythos 5, $ cost ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| BrowseComp 500-cut (Anthropic docs) anthropic | Fable 5, 1-3 runs per setting ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high |
| BrowseComp multi-agent, 1 agent (OpenAI GPT-5.6 post chart) openai | BrowseComp (Multi-Agent chart), 1 agent; code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; 'Ultra is run with 4 agents'; multi-agent cost/tokens include all agents; published 2026-07-09; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| BrowseComp multi-agent, 16 agents (OpenAI GPT-5.6 post chart) openai | BrowseComp (Multi-Agent chart), 16 agents; code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; 'Ultra is run with 4 agents'; multi-agent cost/tokens include all agents; published 2026-07-09; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| BrowseComp multi-agent, 4 agents (OpenAI GPT-5.6 post chart) openai | BrowseComp (Multi-Agent chart), 4 agents; code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; 'Ultra is run with 4 agents'; multi-agent cost/tokens include all agents; published 2026-07-09; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| bug-hunt-bench github-bughuntbench | Claude Code 2.1.251 · 105 planted bugs, blind judge, 3 models × ladder ✓ | sweep | Claude Fable 5.1·max · Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5·max · Claude Fable 5·high · Claude Opus 5·max · Claude Opus 5·xhigh · Claude Opus 5·high · Claude Opus 5·medium · Claude Opus 5.5·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low · Claude Opus 4.8·high · Claude Opus 4.8·max · Claude Sonnet 5·high · Claude Sonnet 5·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · GPT-5.6 Sol·high · Grok 4.5·high · GPT-5.6 Luna·high · Grok 4.6·xhigh · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · Grok 4.6·low · Grok 4.6·medium · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·medium · GPT-5.6 Luna·xhigh · Grok 4.6·high · GPT-5.6 Terra·xhigh · GPT-5.6 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·xhigh · GPT-6 Astra·high · GPT-6 Astra·medium · GPT-6 Astra·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·max · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low · GPT-5.6 Luna·max · GPT-6 Sol·max · GPT-6 Sol·xhigh · GPT-6 Sol·high · GPT-6 Sol·medium · GPT-6 Sol·low · GPT-6 Luna·max · GPT-6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Luna·medium · GPT-6 Luna·low · GPT-6.1 Sol·max · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·high · GPT-6.1 Sol·medium · GPT-6.1 Sol·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · Claude Fable 5.2·max |
| BullshitBench v2 github-bullshitbench | petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓ | sweep | Claude Opus 4.8·nothink · Claude Opus 4.8·xhigh · Claude Sonnet 4.6·high · Claude Sonnet 4.6·nothink · Claude Fable 5.1·low · Claude Opus 4.7·nothink · Claude Fable 5·xhigh · Claude Opus 5·xhigh · Claude Sonnet 5·low · Claude Sonnet 5·max · Claude Opus 5·low · Claude Fable 5.1·max · Claude Fable 5·low · Claude Opus 5.5·low · Claude Opus 5.5·max · Claude Haiku 4.5·nothink · Claude Opus 4.7·max · GLM-5.3·low · GPT-6 Astra·max · GPT-6.1 Sol·max · GLM-5.3·max · GPT-6 Astra·low · GPT-6.1 Sol·low · Claude Sonnet 5.5·low · Gemini 3.5 Flash-Lite·minimal · Gemini 4 Argon·low · Kimi K2.6·nothink · Gemini 4 Argon·high · Grok 4.6·low · Claude Haiku 5.5·max · GPT-6 Sol·max · Claude Haiku 5.5·low · Claude Sonnet 5.5·max · GPT-6 Luna·max · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·high · GPT-6 Sol·nothink · DeepSeek V4.1 Flash·low · GPT-5.6 Terra·max · GPT-6 Luna·nothink · Grok 4.6·xhigh · Grok 4.5·low · Grok 4.5·high · Grok 4.3·low · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.5·xhigh · GPT-5.4·nothink · GPT-5.5·nothink · GPT-5.6 Terra·low · GPT-5.5·low · GPT-5.6 Luna·max · GPT-5.4·xhigh · Grok 4.3·high · GPT-5.2·nothink · GPT-5.5 Pro·xhigh · GPT-5.6 Luna·low · Gemini 3.1 Pro·low · GPT-5.4 mini·xhigh · GPT-5.2·high · DeepSeek V4 Pro 0813 build·max · GPT-5.4 mini·high · Qwen3.8 Max·xhigh · Gemini 3 Pro·high · GPT-5.4 mini·nothink · Qwen3.8-27B·xhigh · Gemini 3.7 Flash·low · DeepSeek V4 Pro 0813 build·low · Gemini 3.6 Flash·minimal · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GLM-5.2·high · Gemini 3.5 Flash·high · GPT-5.4 nano·high · GPT-5.4 nano·nothink · Gemini 3.5 Flash·minimal · GPT-5.4 nano·xhigh · Gemini 3 Flash·high · Grok 4.1 Fast·nothink |
| Buoy: Haiku 5.5 default vs high (256 food-ordering-app jobs) buoy.gg | 256 jobs in Buoy's own test app on iOS Simulators, 1 run each; score = jobs done right %; cost per job from tokens x list price; read 2026-10-15 ✓ | sweep | Claude Haiku 5.5·high · GPT-6 Luna·medium |
| CFG k3s DevOps computingforgeeks | computingforgeeks Sonnet 5.5 review · 3 tasks × 3 runs, raw API single file; score = k3s deploy + HTTP 200 pass rate of the K8s task, cost summed over the 3 tasks ✓ | sweep | Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Sonnet 5.5·max · Claude Sonnet 5·high · GPT-6 Sol·medium |
| Chartography +tools anthropic | Fable 5.1 card p186 · 5 models × sweep low→max, $ cost; Opus 4.8 series joined from Opus 5 card p171 ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max |
| Chartography −tools anthropic | Fable 5.1 card p186 · tools disabled — separate regime, kept out of the effort grid ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max |
| Chartography regraded anthropic | Opus 5.5 card p202 · same transcripts as chartogt, re-graded scores — quality only, costs stay in chartogt ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| claude-effort-bench github-effort-bench | Claude Code 2.1.206 · 3 multi-file tasks × 5 reps, Opus 5 & Sonnet 5 × medium→xhigh, CLI cost (Sonnet ×2/3), deterministic verifiers; ~30 % silent no-op runs scored 0 ✓ | sweep | Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| claudecode_lab · invoice check note-claudecode_lab | note.com 29 Sep · Claude Code 2.1.284 -p, Sonnet 5.5 and Opus 5.5 × low/medium/high/max, 7/7 everywhere (saturated), cost from Claude Code ✓ | sweep | Claude Sonnet 5.5·medium · Claude Sonnet 5.5·max |
| Cookbook cost optimization anthropic | Anthropic cookbook cost_optimization.ipynb · 10-claim agent ×2 trials, Opus 5 & Sonnet 5 low→high, Haiku 4.5; list-price cost from usage ✓ | sweep | Claude Opus 5·high · Claude Opus 5·medium · Claude Opus 5·low · Claude Sonnet 5·high · Claude Sonnet 5·medium · Claude Sonnet 5·low |
| Corpus defect sweep (Anthropic docs) anthropic | 21.6M-token corpus, F1, per episode ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high |
| CursorBench anthropic | Sonnet 5 card p118 · 5 models × sweep low→max, $ cost, scores printed ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh |
| CursorBench 3.2 cursorbench | Cursor 3.2 · matched effort, $ cost ✓ (vendor benchmark) | sweep | Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Opus 5·low · Claude Opus 5·medium · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Grok 4.6·xhigh · GPT-5.6 Sol·max · Grok 4.6·medium · Gemini 3.8 Flash·medium · GPT-5.6 Terra·max · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·high · GPT-5.6 Luna·max · Grok 4.6·low · Kimi K3·high · GPT-5.6 Terra·xhigh · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·high · GPT-5.6 Terra·high · GPT-5.6 Sol·low · Kimi K3·low · GPT-5.6 Terra·low · GPT-5.6 Luna·low · Grok 4.6·high · Gemini 3.8 Flash·high · Kimi K3·max · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Grok 4.5·high |
| CursorBench 4.0 cursorbench | cursor.com payload · v4.0 long-horizon suite, Opus 5/5.5, Fable 5.1, Sonnet 5/5.5 × low→max, measured tokens × list incl. cache ✓ | sweep | Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Grok 4.6·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·xhigh · Gemini 3.8 Flash·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·low · Grok 4.7·xhigh · Grok 4.7·medium · Grok 4.7·low · Gemini 3.8 Flash·high · GPT-5.6 Sol·medium · Grok 4.7·high · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low · GLM-5.3·max · GPT-5.6 Terra·max · GLM-5.3·high · GLM-5.3-Flash·max · GPT-5.6 Luna·max · GPT-5.6 Terra·xhigh · GLM-5.3·low · GPT-5.6 Luna·xhigh · GLM-5.3-Flash·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high · GPT-5.6 Terra·medium · GLM-5.3-Flash·low · GPT-5.6 Terra·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·low · Claude Fable 5.2·max · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·medium · Claude Fable 5.2·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low · Gemini 4 Hydrogen·xhigh · Gemini 4 Hydrogen·high · Gemini 4 Hydrogen·medium · Gemini 4 Hydrogen·low |
| CyberQ security short answer cyberq | Opus 5.5 ladder + Opus 5, no score ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Opus 5·high |
| CyberQ structured extraction cyberq | Opus 5.5 ladder + Opus 5, no score ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Opus 5·high |
| Databricks · Claude Code databricks | Opus 4.8 high/xhigh/max + Sonnet 5/4.6, Haiku 4.5 ✓ | sweep | Claude Opus 4.8·high · Claude Opus 4.8·max · Claude Opus 4.8·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.5·xhigh |
| DeepResearch futuresearch | Deep Research Bench · low/high ✓ | sweep | Gemini 3 Flash·low · Gemini 3 Flash·high |
| DeepResearch Bench II (Anthropic docs) anthropic | 50-task subset, 33-task basis, customer-billed ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high |
| DeepSearchQA anthropic | Opus 4.7 card p200 · Opus4.7 vs Sonnet4.6, sweep low→max ✓ | sweep | Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·max · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high |
| DeepSearchQA (O5) anthropic | Opus 5 card p161 · 980k budget, 4 models × sweep low→max, $ cost ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| DeepSeek V4.1 tech report effort curve: AIME 2026 deepseek | AIME 2026, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓ | sweep | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high |
| DeepSeek V4.1 tech report effort curve: Apex 2025 Shortlist deepseek | Apex 2025 Shortlist, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓ | sweep | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max |
| DeepSeek V4.1 tech report effort curve: DeepSWE v1.1 DeepSeek Harness Minimal deepseek | DeepSWE v1.1 DeepSeek Harness Minimal, deepseek-v4.1-flash at reasoning effort 25..100 (Fig11); Pass@1 ; harness DeepSeek-Harness-minimal; max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓ | sweep | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max |
| DeepSeek V4.1 tech report effort curve: HLE-Text deepseek | HLE-Text, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓ | sweep | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max |
| DeepSeek V4.1 tech report effort curve: IMO-AnswerBench deepseek | IMO-AnswerBench, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓ | sweep | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max |
| DeepSeek V4.1 tech report effort curve: LiveCodeBench deepseek | LiveCodeBench, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓ | sweep | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max |
| DeepSeek V4.1 tech report effort curve: SimpleQA-Verified deepseek | SimpleQA-Verified, deepseek-v4.1-flash at reasoning effort 25..100 (Fig12); Pass@1 ; harness none(reasoning); max_steps 500, N=8 (DeepSWE) / N=3 (Terminal-Bench) per Table 4 note; digitised exactly from the PDF vector paths (calibration residual < 1e-4); read 2026-10-19 ✓ | sweep | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max |
| DeepWideSearch (Anthropic docs) anthropic | Fable 5, 220 questions, 3 runs ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high |
| DRACO anthropic | Fable 5 card p271 · Fable=Mythos 5, sweep low→max, $ cost ✓ | sweep | Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max |
| DRACO (F5.1) anthropic | Fable 5.1 card p179 · 980k budget, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh |
| DRACO (O5.5) anthropic | Opus 5.5 card p187 · 980k budget, 3 models × low→max, $ cost, scores printed; a new run, not the F5.1-card one ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| DRACO (Opus 5) anthropic | Opus 5 card p162 · 980k budget, sweep low→max; Fable=Mythos 5, $ cost ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| FrontierCode main cognition | Cognition leaderboard JSON · v1.1 main (100 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · Grok 4.5·low · Grok 4.5·medium · Grok 4.5·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·high · Grok 4.7·xhigh · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · Claude Opus 4.7·xhigh · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Gemini 3.6 Flash·low · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·high · GPT-5.4 mini·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash·high · GLM-5.3·max · GLM-5.3-Flash·max · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| FrontierCode v1 cognition | Sonnet 5 card p117 · 4 models × sweep low→max, $ cost, scores printed ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · Claude Opus 4.7·xhigh · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·high · GPT-5.4 mini·xhigh |
| FrontierCode-D anthropic | Fable 5 card p257 · Fable=Mythos 5, sweep low→max, $ cost ✓ | sweep | Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh |
| FrontierCode-Ext cognition | Cognition leaderboard JSON · v1.1 Extended (150 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·high · Grok 4.7·xhigh · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 4.7·xhigh · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·high · GPT-5.4 mini·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash·high · GLM-5.3·max · GLM-5.3-Flash·max · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| FrontierFinance arxiv-2608.11683 | arXiv 2608.11683 Fig. 9 · Finance Agent v2 harness, Opus 4.8 low→xhigh, API cost/query (vector-read) ✓ | sweep | Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh |
| GDPval (Anthropic docs) anthropic | Fable 5, 1 run, cost incl. grading ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high |
| George Liu · Opus 5 vs 5.5 ai.georgeliu | Claude Code 10-prompt suite · low→max · IFEval /9 ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| Haiku 5.5 launch page: GDPval-AA v2.1, accuracy vs cost anthropic | anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max |
| Haiku 5.5 launch page: Humanity's Last Exam (no tools), accuracy vs cost anthropic | anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| Haiku 5.5 launch page: OSWorld 2.1 (offline subset), accuracy vs cost anthropic | anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max |
| Haiku 5.5 system card: HealthBench by effort, 5 levels anthropic | 1 run/level below max, max = mean of 5 runs; read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low |
| Haiku 5.5 system card: HealthBench Professional (length-adjusted) by effort, 5 levels anthropic | 5 runs/level except 13 repeat runs at max on a different day (mean 64.2% vs reported 64.8%, see confound); read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·max · Claude Haiku 5.5·high |
| Haiku 5.5 system card: OSWorld 2.1 (offline subset) strict pass rate by effort, 5 levels (digitised) anthropic | strict pass rate (full-credit, binary) vs effort; cost x-axis identical to the partial-credit OSWorld chart and to the launch-page OSWorld chart (same underlying runs); only the max point (37.1%) is independently stated in text (p.128), the other four come only from this chart; read 2026-10-08 ✓ | sweep | GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max |
| Haiku 5.5 system card: PhysicianBench by effort, 5 levels, pass rate anthropic | 5 attempts/task at each level; read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| haiku-thinking-bench: does thinking-off make Haiku 5.5 skip a needed handoff (24 cases x 10 runs, 80 handoff runs) github-haiku-thinking-bench | 17tayyy/haiku-thinking-bench README (Oct 2026); mechanical grading; score = handoffs made correctly of 80 (100 - missed%); cost = cost per conversation (tokens x list price, no caching); read 2026-10-15 ✓ | sweep | Claude Haiku 4.5·nothink · Claude Haiku 5.5·nothink · Claude Haiku 5.5·low · Claude Haiku 5.5·medium |
| harnesseval github-harnesseval | Claude Code · 6 PRs common to every cell, CLI-metered cost ✓ | sweep | Claude Fable 5.1·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · Claude Opus 5·high · GLM-5.3·low · GLM-5.3·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · Claude Sonnet 5·high · GLM-5.3-Flash·low · GLM-5.3-Flash·high |
| HealthBench Pro (adj.) anthropic | Sonnet 5.5 card p138 · length-adjusted, Sonnet 5.5 low→max + 3 at max, score-only ✓ | sweep | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Fable 5.1·max |
| HellaSwag effort (interface-gap) arxiv-2609.08861 | direct API · budget_tokens low/med/high, Sonnet4.6 & Haiku4.5 ✓ | sweep | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high |
| HLE (Opus 5) anthropic | Opus 5 card p157 · HLE tools, 4 models × sweep low→max, $ cost ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| HLE no-tools (F5.1) anthropic | Fable 5.1 card p178 · HLE without tools, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·medium · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| HLE no-tools (S55) anthropic | Sonnet 5.5 card p120 · 980k budget, 3 models × low→max, $ cost (perfect caching), scores printed ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| HLE sweep anthropic | Opus 4.8 card p203 · HLE tools, sweep low→max ✓ | sweep | Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·max |
| HLE tools (F5.1) anthropic | Fable 5.1 card p177 · HLE with tools, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·medium |
| HLE tools (O5.5) anthropic | Opus 5.5 card p185 · HLE with tools, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| ibragim.dev · Claude ibragim-dev | Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Claude Code 2.1.281, effort per row, pass@1 and $/task; board of 28 Sep ✓ | sweep | Claude Opus 5.5·high · Claude Sonnet 5·high |
| ibragim.dev · Pi ibragim-dev | Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Pi 0.85.1, effort per row, pass@1 and $/task; board of 28 Sep ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5·high · Claude Sonnet 5·high · GPT-6 Luna·xhigh · GPT-6.1 Sol·medium · GPT-6 Sol·medium · GPT-6.1 Sol·high · GPT-6 Astra·low · GPT-6 Luna·medium · GLM-5.3-Flash·max · GPT-5.6 Sol·medium · GPT-6 Luna·high · DeepSeek V4.1 Flash·max · GPT-5.6 Luna·medium · GPT-5.6 Terra·medium · Qwen3.8-27B·xhigh |
| InfluMatch Set 1 (11 queries) P@5, reason on all 50 candidates arxiv-2607.05968 | KOL search P@5 %, Kimi-K2.6 as reason scorer think vs no-think; tokens per query from Table 10 (no USD); paper 2026-07-07; read 2026-10-20 ✓ | sweep | Kimi K2.6·think · Kimi K2.6·nothink |
| InfluMatch Set 1 P@5, SimPO rerank then reason top-10 arxiv-2607.05968 | KOL search P@5 %, Kimi-K2.6 as reason scorer think vs no-think; tokens per query from Table 10 (no USD); paper 2026-07-07; read 2026-10-20 ✓ | sweep | Kimi K2.6·think · Kimi K2.6·nothink |
| InfluMatch Set 2 (31 queries) P@5, reason top-10 after SimPO rerank arxiv-2607.05968 | KOL search P@5 %, Kimi-K2.6 as reason scorer think vs no-think; tokens per query from Table 10 (no USD); paper 2026-07-07; read 2026-10-20 ✓ | sweep | Kimi K2.6·think · Kimi K2.6·nothink |
| Internal coding, 370 repo tasks (Anthropic docs) anthropic | plain API agent, customer-billed per attempt ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high |
| jev-effort github-jev-effort | Claude Code 2.1.280 · Opus 5.5 baseline arm, high vs max fixed effort, 6 coding tasks x2 runs ✓ | sweep | Claude Opus 5.5·high · Claude Opus 5.5·max |
| kayben plan review github-kaybenleroll | claude -p no tools, CC 2.1.282 · 13 seeded defects, Opus 5.5 & Sonnet 5 × medium/high/xhigh, n=10, effort verified in transcripts ✓ | sweep | Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| kayben plan review f51 github-kaybenleroll | same doc, cap10 prompt · Fable 5.1 low→max, n=5 ✓ | sweep | Claude Opus 5.5·max · Claude Sonnet 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max |
| Lisbon bike-rental website, Sonnet 5.5 effort ladder (Mahe Delich video) youtube-mahe-delich | Claude Code /cost; same prompt; empty folder; one try per level; 2026-09-30; read 2026-10-18 ✓ | sweep | Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| llm-regression 2026-10-02 Opus 5.5 high vs xhigh (pass rate %) nikoamoretti-llm-regression | nikoamoretti/llm-regression data.json: 2026-10-02 first live runs, product suite, Claude Code, high (32 attempts) vs xhigh (28 attempts, 3 infra_fail); strict hidden-test pass rate over graded attempts; read 2026-10-20 ✓ | sweep | Claude Opus 5.5·xhigh · Claude Opus 5.5·high |
| Low-poly forest in Blender, Haiku 5.5 effort ladder (DopeShit video) youtube-dopeshit | Claude Code, Haiku 5.5 effort set per run; author grades 4 criteria (10 each) = /40; 2026-10-10; read 2026-10-18 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| m_izumo Opus 5.5 review sub-agents on a personal project (15 sections) qiita-m_izumo | Claude Code sub-agents (20 x high; 22 x xhigh); defects verified by main session; coverage = share of verified real defects found by one reviewer; read 2026-10-15 ✓ | sweep | Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| MARROWLIGHT one-page site, Opus 5.5 effort ladder (The Useful Stack video) youtube-usefulstack | Claude Code 2.1.285 --effort; auto-scored /100; redone runs only; one run per setting; 2026-09-30; read 2026-10-18 ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh |
| MCPMark eval-sys/mcpmark | eval-sys/mcpmark legacy board · 5 MCP servers (FS/GitHub/Notion/Playwright/Postgres), Sonnet 4 low/default/high + Opus 4.1/Sonnet 4.5/Opus 4.5, Cost-per-run; low>high cost inversion kept as measured ✓ | sweep | Gemini 3 Pro·high · GPT-5.2·high |
| MERC core-10 github-merc-bench | Claude Code -p, no tools · 10 graded tasks, 6 models × low→max, n=2; usage × list (5m cache-write); scores 94–98/98 → cost only ✓ | sweep | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Claude Haiku 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Haiku 5.5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·xhigh · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·low · Claude Opus 5.5·max · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·max · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·max · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·xhigh |
| nnakapa #35 Copilot zenn-nnakapa | GitHub Copilot CLI 1.0.79 · same 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, billed cost ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| nnakapa #36 Copilot zenn-nnakapa | GitHub Copilot CLI 1.0.80 · same 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, billed cost ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high |
| nnakapa QCD #20 zenn-nnakapa | Claude Code 2.1.157 · 3 DB tasks × 10 runs, Opus 4.7 & Opus 4.8 × low→xhigh ✓ | sweep | Claude Opus 4.7·xhigh · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh |
| nnakapa QCD #24 zenn-nnakapa | Claude Code 2.1.170 · 3 DB tasks × 10 runs, Fable 5 & Opus 4.8 × low→xhigh ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh |
| nnakapa QCD #29 zenn-nnakapa | Claude Code 2.1.220 · 3 DB tasks × 10 runs, Opus 5, Opus 4.8 & Fable 5 × low→xhigh, CLI-reconciled cost, hidden-test gate ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh |
| nnakapa QCD #34 zenn-nnakapa | Claude Code 2.1.226 · 3 DB tasks × 10 runs, Sonnet 5 & Opus 5 × low→xhigh, CLI-reconciled cost ($2/$10 Sonnet), hidden-test gate ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh |
| nnakapa QCD #35 CC zenn-nnakapa | Claude Code 2.1.229 (1M ctx) · 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, CLI-reconciled cost ($2/$10) ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| nnakapa QCD #37 zenn-nnakapa | Claude Code 2.1.258 · 3 DB tasks × 10 runs, Opus 5 & Fable 5.1 × low→xhigh, CLI-reconciled cost, hidden-test gate ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh |
| ObviousBench 0.2 benchmarklist | inspect-ai · 144 items ×3 epochs, 3 models × full ladder, measured tokens ✓ | sweep | Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Haiku 5.5·nothink · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·max · Claude Opus 4.8·nothink · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Sonnet 4.6·nothink · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · GPT-5.3-Codex·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · Kimi K2 Thinking·solo · Kimi K2.7 Code·solo · GPT-5.2·nothink · GPT-5.2·low · GPT-5.2·medium · GPT-5.2·high · GPT-5.2·xhigh · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·nothink · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·high · GPT-5.4 mini·xhigh · GPT-5.4 nano·nothink · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·high · GPT-5.4 nano·xhigh · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·nothink · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·nothink · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · Qwen3.7 Flash·nothink · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Qwen3.8 Flash·nothink · Qwen3.8 Max·low · Qwen3.8 Max·xhigh · Qwen3.8 Max 0902 build·low · Qwen3.8 Max 0902 build·medium · Qwen3.8 Max 0902 build·xhigh · Grok 4.5·low · Grok 4.5·medium · Grok 4.5·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high |
| Occam bench, no plugin github-occam | Claude Code headless · 18 procedural tasks, hidden verifier · Sonnet 5.5 & Opus 5.5 at max/medium + Haiku 4.5 ✓ | sweep | Claude Opus 5.5·max · Claude Opus 5.5·medium · Claude Sonnet 5.5·max · Claude Sonnet 5.5·medium |
| Opus 5.5 AoE2 build github-barty-opus55 | Claude Code 2.1.281 · one long game-build prompt, Opus 5.5 × low→max, 1 run, CLI cost, no score → cost only ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Sonnet 5.5·max · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6 Astra·high · GPT-6 Astra·xhigh |
| OSWorld anthropic | Anthropic/AA · sweep low→max ✓ | sweep | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high |
| OSWorld 2.0 anthropic | Opus 5 card p174 · price vs perf, 4 models; effort inferred from cost order ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| OSWorld 2.0 (F5.1) anthropic | Fable 5.1 card p190 · partial-credit price/perf, 3 models; effort inferred from cost order ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max |
| OSWorld 2.0 board (Opus 5) osworld-v2-board | osworld-v2.xlang.ai official JSON · v2.1 full 108 tasks, batched, 500 steps, Opus 5 low→max, output tokens/task, no cost ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| OSWorld 2.0 board v08.08 (Opus 5) osworld-v2-board | osworld-v2.xlang.ai official JSON · release v2026.08.08, 7-run avg, Opus 5 low→max, score only ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Sol·max |
| OSWorld 2.0 code_mode=true (OpenAI GPT-5.6 post chart) openai | OSWorld 2.0, code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| OSWorld 2.1 (O5.5) anthropic | Opus 5.5 card p207 + Sonnet 5.5 card p131 · v2.1 files of 10 Sep, partial credit, 3 models × 5 efforts (effort from cost order) + Sonnets at max ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Sonnet 5·max |
| OSWorld 2.1 strict anthropic | Sonnet 5.5 card p132 · strict pass, Opus 5.5 × 5 (effort from cost order) + Sonnets at max; other Sonnet points held ✓ | sweep | Claude Sonnet 5.5·max · Claude Sonnet 5·max |
| OSWorld eff. (4.7) anthropic | Opus 4.7 card p209 · pass@1 vs output tokens, 3 models × low→max ✓ | sweep | Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·max · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high |
| OSWorld sweep anthropic | Opus 4.8 card p222 · sweep low→max, output tokens ✓ | sweep | Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·max |
| oubakiou godot-llm-gamebench: Opus 5.5 child, 5 efforts, parent fixed opus-5.5@medium zenn-oubakiou | Claude Code CLI child at claude-opus-5-5, parent claude-opus-5-5@medium fixed; Godot 4.4.1 'Ember Warden' mini-game vs private spec, scored by 57 hidden tests; n=3 reps per effort; zenn.dev/oubakiou/articles/fa3b3914e081c8; read 2026-10-05 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| pelican-bicycle-sigma.vercel.app — Haiku 5.5 effort sweep (SVG pelican-on-bicycle) pelican-bicycle-sigma | independent site, one Claude model (Haiku 5.5) at five efforts = five couples per RULES.md; cost is the page's own $ figure computed from a usage panel (not a billed invoice) -> list-price-cost; page's own caveat: "Usage numbers are transcribed from each run's usage panel" (manual transcription, not raw API JSON); read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| PetriBench arxiv-2609.19883 | arXiv 2609.19883 · 4,800 Petri-net questions, raw API adaptive thinking + output_config.effort, Opus 5 M/H/XH + Sonnet 5 M/H, measured tokens × list ✓ | sweep | GPT-5.6 Sol·xhigh · Claude Opus 5·xhigh · Claude Opus 5·high · GPT-5.6 Sol·high · Grok 4.6·xhigh · Grok 4.6·high · Grok 4.6·medium · Gemini 3.8 Flash·high · Claude Opus 5·medium · Gemini 3.1 Pro·high · Grok 4.5·high · GPT-5.6 Terra·xhigh · Grok 4.5·medium · Gemini 3.8 Flash·medium · GPT-5.6 Sol·medium · Gemini 3.1 Pro·medium · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·high · GPT-5.6 Luna·high · Claude Sonnet 5·high · Grok 4.3·medium · GPT-5.6 Terra·medium · Claude Sonnet 5·medium · Claude Haiku 4.5·nothink · GPT-5.6 Luna·medium |
| PhysicianBench anthropic | Sonnet 5.5 card p138 · Sonnet 5.5 low→max labelled + 4 models at max, score-only (time axis) ✓ | sweep | Claude Opus 5·max · Claude Sonnet 5·max |
| Playcode MacBook SVG playcode | Playcode · one-shot MacBook SVG, high/xhigh/max per model, provider-billed cost (quality ordinal, not scored) ✓ | sweep | GLM-5.3-Flash·low · Claude Haiku 5.5·low · GPT-6 Luna·medium · Claude Haiku 5.5·medium · GPT-5.6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · Claude Haiku 5.5·high · GPT-5.6 Luna·high · Claude Haiku 5.5·xhigh · GLM-5.3-Flash·high · GLM-5.3·low · GPT-5.6 Luna·xhigh · GPT-6 Luna·max · Claude Sonnet 5·high · GPT-5.6 Luna·max · Grok 4.6·high · Gemini 3.6 Flash·medium · GPT-6 Sol·medium · Gemini 3.8 Flash·medium · Claude Opus 4.8·high · GPT-5.6 Terra·high · GPT-6 Sol·high · Claude Opus 4.8·xhigh · Claude Sonnet 4.6·high · Claude Sonnet 5·xhigh · Gemini 4 Argon·medium · Gemini 4 Argon·high · Kimi K3·high · GPT-6 Sol·xhigh · Gemini 4 Argon·low · GPT-5.6 Terra·xhigh · Claude Opus 4.8·max · GPT-5.6 Sol·high · GLM-5.3·high · Claude Fable 5·high · GLM-5.3·max · GPT-6 Sol·max · Claude Opus 5.5·high · GPT-5.6 Terra·max · GPT-5.6 Sol·xhigh · Kimi K3·max · Claude Opus 5·high · Claude Fable 5·xhigh · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·xhigh · Claude Opus 5·xhigh · Claude Sonnet 5·max · Claude Opus 5.5·xhigh · Claude Opus 5·max · Claude Fable 5·max · GPT-6 Astra·max · Claude Opus 5.5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·max |
| PostTrainB. arxiv-2603.08640 | papier · medium/high ✓ | sweep | GPT-5.3-Codex·high · GPT-5.3-Codex·medium · Kimi K2 Thinking·solo |
| Qiita yunany90 Opus 5.5 effort sweep, brick-breaker single index.html qiita-yunany90 | Claude Code (Windows; Max 20x); same request 5 times, only effort changed; cost = ccusage API-equivalent; read 2026-10-15 ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| QuoteBench arxiv-2608.13547 | arXiv 2608.13547 · 56 frozen tasks, provider-reported output tokens, 3 models × ladder ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Qwen3.5-27B·nothink |
| Ramen Bench github-ramenbench | not-stbenjam/ramen-bench · one-shot creative-coding HTML, Sonnet 5.5, Opus 5.5, Opus 5, Fable 5.1 × low→max, Claude Code run.json, cost only ✓ | sweep | Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Claude Haiku 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Haiku 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·low · Claude Opus 5.5·max · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·max · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·xhigh · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · Gemini 4 Argon·low · Gemini 4 Argon·medium · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·max · GPT-5.6 Luna·medium · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·xhigh · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·max · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·max · GPT-6 Luna·medium · GPT-6 Luna·xhigh · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·max · GPT-6 Sol·medium · GPT-6 Sol·xhigh · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·max · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · Grok 4.5·high · Grok 4.5·low · Grok 4.5·medium · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·high · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·xhigh |
| Reddit 1x0iu5x (dergachoff): Haiku 5.5 research sub-agent at three efforts vs DeepSeek reddit-dergachoff | 4 real research briefs, Haiku 5.5 low/medium/high vs DeepSeek V4.1 Flash (low), Opus blind A/B judge both orders; score = briefs won vs DeepSeek of 4 (ties counted as not won); cost per run as self-reported (scope unstated); read 2026-10-15 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high |
| Reddit: pelican-on-a-bicycle SVG, Opus 5.5 effort sweep reddit-1wrdd57 | one-line prompt 'Write HTML code that displays a 2D animation of a pelican riding a bicycle using SVG graphics...', fresh session each, author's own agent setup (pi with custom skills); n=1 per level, times/costs from session logs, no numeric quality score (qualitative only); read 2026-10-05 ✓ | sweep | Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| renchris review f51 github-renchris-infra | claude -p no tools, CC 2.1.260 (64K cap) · same 9-brief corpus, Fable 5.1 low→xhigh (max incomplete), 3 blind judges ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh |
| renchris review o55 github-renchris-infra | claude -p no tools, CC 2.1.280 · 9 code briefs / 36 anchored defects, Opus 5.5 low→max + Opus 5 @high, 3 blind judges; CLI list cost ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Opus 5·high |
| renkon40 · 12 editable PowerPoint slides in a given style note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, 12 editable PowerPoint slides in a given style; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | GPT-6 Sol·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5.5·xhigh |
| renkon40 · 3-minute talk slides note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, 3-minute talk slides; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Claude Fable 5.1·max · GPT-6 Astra·max · GPT-6 Sol·max · GPT-6 Luna·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| renkon40 · dashboard from an AI-secretary log note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, dashboard from an AI-secretary log; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | GPT-6 Astra·high · Claude Opus 5.5·high · GPT-6 Sol·high · Claude Fable 5.1·high · GPT-6 Luna·high · Claude Sonnet 5.5·high · Claude Sonnet 5.5·max |
| renkon40 · Note article from an experiment log note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, Note article from an experiment log; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high · Gemini 3.8 Flash·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Sonnet 5.5·high · Claude Sonnet 5.5·max |
| renkon40 · short video note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, short video (60 s from a 25-min YouTube talk); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Claude Opus 5.5·max · GPT-6 Astra·max · Claude Fable 5.1·max · GPT-6 Sol·max · Gemini 3.8 Flash·high · GPT-6 Luna·max · Claude Sonnet 5.5·max |
| renkon40 · YouTube-channel homepage note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, YouTube-channel homepage (no skill); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | GPT-6 Sol·high · GPT-6 Luna·high · GPT-6 Astra·high · Claude Fable 5.1·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Sonnet 5.5·max |
| retort exp-49 github-retort | Claude Code · one task, 2 models × low→max, n=3; repo states its own quality saturation → cost only ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·max · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·max |
| retort exp-55 brazil go github-retort | Claude Code · brazil-soccer-mcp Go, Opus 5 low/high/xhigh/max (medium has no cost), n=1 ✓ | sweep | Claude Opus 5·low · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5.5·low |
| retort exp-55 brazil py github-retort | Claude Code · brazil-soccer-mcp Python, Opus 5 low→max, n=1 ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5.5·low |
| retort exp-55 go github-retort | Claude Code · rest-api-crud Go, Opus 5 low→max, n=2 (max n=1); coverage 1.0 → cost only ✓ | sweep | Claude Opus 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh |
| retort exp-55 py github-retort | Claude Code · rest-api-crud Python, Opus 5 low→max, n=2 (max n=2); coverage 1.0 → cost only ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| retort exp-74 go github-retort | Claude Code · rest-api-crud in Go, Opus 5.5 × low→max, n=3; requirement coverage 1.0 everywhere → cost only ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| retort exp-74 py github-retort | Claude Code · rest-api-crud in Python, Opus 5.5 × low→max, n=3; requirement coverage 1.0 everywhere → cost only ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| retort exp-79 go isolated github-retort | Claude Code (isolated, MCP disabled) · rest-api-crud in go, Opus 5.5 × low/medium/high/default, n=3; requirement coverage 1.0 everywhere → cost only; read 2026-10-05 ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| retort exp-79 python isolated github-retort | Claude Code (isolated, MCP disabled) · rest-api-crud in python, Opus 5.5 × low/medium/high/default, n=3; requirement coverage 1.0 everywhere → cost only; read 2026-10-05 ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| RHI (Sonnet 4.6) arxiv-2607.15524 | arXiv 2607.15524 Fig. 5b · 30 ML tasks, cost normalized to sonnet-4.6-high ✓ | sweep | Claude Sonnet 4.6·high |
| SEC-Bench Pro multi-agent, 1 agent (OpenAI GPT-5.6 post chart) openai | SEC-Bench Pro (Multi-Agent chart), 1 agent; code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; 'Ultra is run with 4 agents'; multi-agent cost/tokens include all agents; published 2026-07-09; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| SEC-Bench Pro multi-agent, 16 agents (OpenAI GPT-5.6 post chart) openai | SEC-Bench Pro (Multi-Agent chart), 16 agents; code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; 'Ultra is run with 4 agents'; multi-agent cost/tokens include all agents; published 2026-07-09; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| SEC-Bench Pro multi-agent, 4 agents (OpenAI GPT-5.6 post chart) openai | SEC-Bench Pro (Multi-Agent chart), 4 agents; code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; 'Ultra is run with 4 agents'; multi-agent cost/tokens include all agents; published 2026-07-09; read 2026-10-19 ✓ | sweep | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| Self-conditioning test: 100 planted turns, 100% wrong totals (muninn.austegard) muninn-austegard | ICLR-2026 self-conditioning re-run, 50-word dictionary, claude -p --resume, 20 sessions per cell, Haiku 5.5; read 2026-10-18 ✓ | sweep | Claude Haiku 5.5·high · Claude Haiku 5.5·low |
| serotoninmax Haiku 5.5 effort sweep, tagging 146 Japanese book blurbs x 91 questions zenn-serotoninmax | API claude-haiku-5-5 max_tokens 32000, output_config.effort; cached 6.4K-token system prompt; per-item cost = measured tokens x list price; Jev reference not Claude; read 2026-10-15 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·max |
| Simon Willison: pelican-riding-a-bicycle SVG test, own measured cost simonwillison | llm CLI (llm-anthropic plugin), own measured spend (not list-price computation), n=1 per point, quality is ordinal/qualitative (no numeric score) -> cost-only; all 5 efforts same session per h4's gist; read 2026-10-08 ✓ | sweep | Claude Haiku 5.5·low · Claude Haiku 5.5·max |
| Sonar Java sonarsource | Sonar leaderboard JSON · 4,444 Java tasks, single-shot, measured tokens × list price; Opus 5.5 medium/high ✓ | sweep | Claude Opus 4.7·high · Claude Opus 4.8·high · Claude Opus 5·high · Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Sonnet 4.6·high · Claude Sonnet 5.5·high · Gemini 3 Flash·high · Gemini 3 Pro·high · Gemini 3.1 Flash-Lite·high · Gemini 3.1 Pro·high · Gemini 4 Argon·high · GPT-5.1·high · GPT-5.1·medium · GPT-5.2·high · GPT-5.4·high · GPT-5.4·medium · GPT-5.4 mini·medium · GPT-5.4 nano·medium · GPT-5.5·low · GPT-5.5·medium · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Astra·medium · GPT-6 Sol·high · GPT-6 Sol·medium · GPT-6.1 Sol·medium |
| SWE Refactor arxiv-2608.23564 | arXiv 2608.23564 · 20 whole-repo migrations, Claude Code, measured API spend, 2 models × ladder ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max |
| SWE-bench Pro subset (Anthropic docs) anthropic | Anthropic-internal 478-problem subset, customer-billed cost ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·high |
| SWE-Effort 25 arxiv-2608.01347 | arXiv 2608.01347 App. N · 25 hard SWE-bench Verified, Claude Code 2.1.220, Opus 5 & Sonnet 5 × low/high/xhigh, billed cost ✓ | sweep | Claude Opus 5·low · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Sonnet 5·low · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| SWE-Pro (Fable) anthropic | Fable 5 card p255 · Fable=Mythos 5, sweep low→xhigh, $ cost ✓ | sweep | Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh |
| SWE-Pro sweep anthropic | Opus 4.8 card p196 · sweep low→max, output tokens ✓ | sweep | Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·max |
| Synthorai (ja mirror): 13 single-shot answer tasks, Sonnet 5.5 / Sonnet 5 / Opus 5.5 by effort synthorai | native Messages API via gateway, 13 tasks x3 reps x6 efforts, cost computed from list price; qiita.com/Synthorai/items/98e776a66f483f3ceeaf (mirror of synthorai.io/ja/blog/claude-sonnet-5-5-cost/); read 2026-10-05 ✓ | sweep | Claude Sonnet 5.5·high · Claude Sonnet 5·high · Claude Sonnet 5.5·low · Claude Sonnet 5·low · Claude Opus 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5·xhigh · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·max · Claude Sonnet 5·max · Claude Opus 5.5·max · GPT-6.1 Sol·medium · GPT-6.1 Sol·low · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max |
| Synthorai multi-hop tool loop (4 Qs ×12) synthorai | Opus 5.5 default/low/max vs Opus 5 default ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·max · Claude Opus 5·high |
| Synthorai single-shot arithmetic (13 tasks ×3) synthorai | Opus 5.5 and Opus 5 full ladders + default, API list-price cost; accuracy ≥97 % everywhere ✓ | sweep | Claude Opus 5·high · Claude Opus 5.5·low · Claude Opus 5·low · Claude Opus 5.5·medium · Claude Opus 5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5·xhigh · Claude Opus 5.5·max · Claude Opus 5·max |
| Tafla benchmark 2026-10-08: everyday calculations, default vs low effort tafla.is | 12 questions x 5 runs x (question + follow-up), live agent with GRID calculation engine; score = correct boards of 120 turns; cost per run (60 runs per lane); read 2026-10-15 ✓ | sweep | GPT-6 Luna·medium · Claude Sonnet 5.5·high · GPT-6.1 Sol·medium · GPT-5.6 Terra·medium · Claude Haiku 5.5·low · GPT-6 Luna·low · Claude Sonnet 5.5·low · GPT-6.1 Sol·low · GPT-5.6 Terra·low |
| tank_ai wareki converter note-tank_ai | note.com 29 Sep · Claude Code subagents at fixed model × effort, n=1, 38 auto-graded tests, score only ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high |
| TB 3.0 (claude.dev chart) anthropic | claude.dev 'Spending your effort' SVG · 70 tasks, Opus 5.5 / Fable 5.1 / Opus 5 / Fable 5 × low→max, median tokens, no USD; Opus 5.5 run 3 wk later, 128k cap ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| TB 3.0 (claude.dev payload) anthropic | claude.dev 'Spending your effort' embedded data · 74 tasks×5, Opus 5 / Fable 5 / Fable 5.1 labelled rungs, median tokens, no USD ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Claude Fable 5.1·max |
| TB 4.0 (Anthropic launch charts) anthropic | Anthropic launch page chart CSVs (Opus 5.5 and Sonnet 5.5 pages) · 5 models × low→max; production safeguards on (fallbacks); Sonnet 5 anomalously low, flagged ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| Towards AI internal writing-Elo board, Sep 25, 2026 reddit-1wr03jc-img2 | 10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; Opus 5.5 six-effort scatter + Fable 5.1 reference point, footer 'Sep 25, 2026'; values are printed point labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓ | sweep | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·max |
| uhyo React proficiency zenn-uhyo | zenn.dev uhyo, round 17 (25 Sep) · 13 React specs × 3 samples, judge pinned to Sonnet 4.6, effort pinned; score only ✓ | sweep | Claude Opus 5.5·max · Claude Opus 5.5·high · Claude Opus 5·max · Claude Opus 5·high · Claude Fable 5·max · Claude Fable 5.1·high · Claude Opus 4.8·max · Claude Fable 5·high · GPT-5.6 Sol·max · GPT-6 Astra·high · GPT-5.6 Sol·high · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Terra·max · GPT-5.6 Terra·high · Claude Sonnet 5.5·high · Claude Sonnet 5.5·max · Claude Haiku 5.5·high · Claude Haiku 5.5·max |
| Vals blog: Vibe Code Bench, Opus 5.5 single agent vals.ai | Claude Code CLI headless, Opus 5.5, single agent, 50 apps, 1 run per setup; score = share of UI tests passed; cost = list-price totals reported by Claude Code per app; read 2026-10-15 ✓ | sweep | Claude Opus 5.5·medium · Claude Opus 5.5·max · GPT-6 Sol·medium · GPT-6 Sol·max |
| Vals blog: Vibe Code Bench, Opus 5.5 team (lead + up to 5 subagents) vals.ai | Claude Code CLI headless, Opus 5.5, team, 50 apps, 1 run per setup; score = share of UI tests passed; cost = list-price totals reported by Claude Code per app; read 2026-10-15 ✓ | sweep | Claude Opus 5.5·medium · Claude Opus 5.5·max · GPT-6 Sol·medium · GPT-6 Sol·max |
| Vals VibeCode vals.ai | Vals AI · Opus 5 sweep low→max (scores), $ cost at leaderboard setting; board entries — snapshot 27 Sep, re-priced ✓ | sweep | DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5·max · Claude Sonnet 5·max · Gemini 4 Carbon·high · GLM-5.2·max · GLM-5.3·max · GPT-5.1-Codex-Max·high · GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max · Kimi K2.7 Code·solo · Claude Fable 5·max · Claude Fable 5.2·high |
| Vals Web Search (finance) vals.ai | vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓ | sweep | Gemini 3.5 Flash·high · Grok 4.5·high · GPT-5.6 Sol·max |
| Vals Web Search (legal) vals.ai | vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓ | sweep | Gemini 3.5 Flash·high · Grok 4.5·high · GPT-5.6 Sol·max |
| vibe-openscad github-vibeopenscad | output_config.effort explicit · 3 models × ladder; quality saturates → cost only ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Fable 5·low · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 4.7·low · Claude Opus 4.8·low · Claude Sonnet 4.6·low · Claude Sonnet 5·low · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·nothink · Claude Opus 4.8·nothink · Claude Sonnet 4.6·nothink · Claude Opus 4.7·xhigh · Claude Opus 4.8·xhigh · Claude Sonnet 5·xhigh · Claude Haiku 4.5·nothink · Claude Sonnet 5·high · Gemini 3.1 Flash-Lite·minimal · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal · Gemini 3.6 Flash·minimal · GPT-5.4·high · GPT-5.4·low · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.5·high · GPT-5.5·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·max · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·xhigh · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·max · GPT-6 Astra·xhigh · Qwen3.8-27B·xhigh |
| vlm-exam github-vlmexam | roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓ | sweep | Claude Fable 5·low · Claude Fable 5·high · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Sonnet 5·low · Claude Sonnet 5·high · Claude Opus 4.8·low · Claude Opus 4.8·high · Claude Opus 5·low · Claude Opus 5·high · Claude Opus 5.5·low · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·high · Gemini 3 Flash·low · Gemini 3 Flash·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high · GPT-5.5·low · GPT-5.5·high · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.6 Sol·low · GPT-5.6 Sol·high · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Sol·low · GPT-6 Sol·high · GPT-6 Luna·low · GPT-6 Luna·high · GPT-6.1 Sol·low · GPT-6.1 Sol·high · Qwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink · GLM-5.3-Flash·low · GLM-5.3-Flash·high · Kimi K2.6·nothink · Kimi K3·high · Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high · DeepSeek V4 Flash 0731 build·high · Gemini 4 Argon·low · Gemini 4 Argon·high |
| VulcanBench CII v4 github-vulcanbench | Claude Code 2.1.280 · 23 legacy binary-parity tasks, Opus 5.5 × low→max, CLI-reported cost; refusal fallback on (0→48 % of replies by Opus 4.8) ✓ | sweep | Claude Opus 5.5·low · Claude Sonnet 5.5·low · GPT-6.1 Sol·low · GPT-6 Astra·low · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6.1 Sol·medium · GPT-6 Astra·medium · Claude Sonnet 5.5·high · GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·xhigh · GPT-6 Astra·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Haiku 5.5·low · Claude Haiku 5.5·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high |
| VulcanBench CII-v4 code_quality (LLM judge) for the held Opus 5.5 ladder github-vulcanbench | github.com/morganlinton/VulcanBench, src/vulcan_gpt61sol-opus55-astra-quality-efforts.csv (commit 3f1bfa61, 2 Oct 2026), same combined-v3.15 Opus 5.5 ladder as group vulcanbench-ciiv4; read 2026-10-05 ✓ | sweep | Claude Opus 5.5·low · Claude Sonnet 5.5·low · GPT-6.1 Sol·low · GPT-6 Astra·low · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6.1 Sol·medium · GPT-6 Astra·medium · Claude Sonnet 5.5·high · GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·xhigh · GPT-6 Astra·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Haiku 5.5·low · Claude Haiku 5.5·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high |
| VulcanBench F-v4 f51 github-vulcanbench | Claude Code 2.1.259–261 · same 23 tasks, Fable 5.1 × low→max, v3.4 judge, per-model list ledger; fallback 7–40 % ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max |
| VulcanBench v3 Opus 5 github-vulcanbench | VulcanBench harness · 23 tasks ×1, Opus 5 low/medium/high, report No.10 (26 Jul) ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high |
| WANDR anthropic | Opus 5.5 card p188 · Perplexity wide-search, offline index, 980k budget, 3 models × low→max, scores printed ✓ | sweep | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max |
| WeirdML epoch-archive | Epoch archive · 3 models × high/max, cost per run ✓ | sweep | GPT-6 Astra·max · Claude Fable 5.1·max · GPT-6 Astra·high · Claude Fable 5.1·high · Claude Fable 5·max · Claude Opus 5·max · Claude Opus 5·high · GPT-5.6 Sol·high · Claude Fable 5·high · GPT-5.6 Sol·max · GPT-5.5·xhigh · GPT-5.5·high · Claude Opus 4.8·xhigh · Kimi K3·max · GPT-5.6 Terra·high · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · Claude Opus 4.7·high · Claude Opus 4.8·medium · Claude Opus 4.7·max · GLM-5.3·max · GPT-5.2·xhigh · Claude Sonnet 5·high · GLM-5.2·high · Grok 4.6·high · GPT-5.5·nothink · DeepSeek V4 Pro 0813 build·max · GPT-5.2·medium · Gemini 3.5 Flash·high · GPT-5.6 Luna·high · GPT-5.1·high · GPT-5.4 mini·high · GPT-5.4·nothink · DeepSeek V4 Flash 0731 build·high · Gemini 3.6 Flash·high · Kimi K2.7 Code·solo · GPT-5.2·nothink · GPT-5.2·low · GPT-5.4 nano·high · Gemini 3.5 Flash-Lite·high · GPT-5.4 nano·nothink · GPT-5.4 mini·nothink |
| WideSearch (Anthropic docs) anthropic | Fable 5 low/medium/default, 3 runs ✓ | sweep | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high |
| Willison SVG 5.1 simonwillison | llm CLI · one fixed SVG prompt, Fable 5.1 low→max, measured tokens; trivial task ✓ | sweep | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max |
| wook3024 CC bench github-wook3024 | Claude Code · 3 tasks × 2 runs, Sonnet 4.6 & Opus 4.6 × medium/high (+ fixed-thinking Opus), CLI cost ✓ | sweep | Claude Sonnet 4.6·high |
| Zapier AutomationB. benchmarklist | AutomationBench 1.0.6 · 657 held-out tasks, strict pass/fail, measured tokens × list ✓ | sweep | Gemini 3.5 Flash-Lite·medium · Gemini 3.5 Flash-Lite·high · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Gemini 4 Argon·high · Gemini 4 Argon·medium · GPT-6 Astra·max · GPT-6 Astra·xhigh · GPT-6 Astra·high · GPT-6 Astra·medium · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · Gemini 3.7 Flash·high · GPT-6 Astra·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · GPT-5.6 Sol·max · GPT-6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·high · GPT-5.6 Terra·max · GPT-6 Sol·low · GPT-6 Luna·max · GPT-5.6 Sol·medium · GPT-5.6 Luna·max · Gemini 3.6 Flash·high · GPT-5.5·xhigh · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·low · GPT-5.6 Terra·xhigh · GPT-6 Luna·high · Gemini 3.5 Flash·high · GPT-5.6 Luna·xhigh · GPT-5.5·high · GPT-6 Luna·xhigh · GPT-5.6 Terra·high · GPT-5.6 Sol·low · GPT-5.4·xhigh · Gemini 3.5 Flash·medium · GPT-5.4·high · GPT-6 Luna·medium · GPT-5.6 Luna·high · GPT-6 Sol·nothink · GPT-5.5·medium · GPT-5.6 Terra·medium · GPT-5.6 Terra·low · GPT-5.4·medium · GPT-5.6 Luna·medium · Kimi K2.7 Code·solo · GPT-5.5·low · GPT-5.6 Luna·low · GPT-5.4·low · GPT-6 Luna·low · Gemini 3.5 Flash·minimal · GPT-6 Luna·nothink · Gemini 3.5 Flash-Lite·low · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Fable 5.2·max · Claude Fable 5.2·xhigh · Claude Fable 5.2·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Hydrogen·high · Gemini 4 Hydrogen·medium · Claude Fable 5.2·low · Claude Fable 5.2·medium · Gemini 4 Carbon·low |
| zenn QCD #27 zenn-nnakapa | Claude Code 2.1.197 · 3 DB tasks × 10 runs, Opus 4.8 & Sonnet 5 × low→xhigh, CLI-reconciled cost ✓ | sweep | Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| deepseek-v41-techreport: deepswe deepseek | DeepSWE v1.1 (Resolved), mini-SWE harness; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness mini-SWE; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | DeepSeek V4.1 Flash·max · Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max |
| deepseek-v41-techreport: gpqa deepseek | GPQA Diamond (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: mathapex deepseek | MathArena Apex (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · Kimi K3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: tb21 deepseek | Terminal-Bench 2.1 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max |
| Deictic ambiguity ablation experiment balanced accuracy arxiv-2609.12162 | Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); ablation experiment; read 2026-10-19 ✓ | xgen | GPT-5.2·xhigh · Gemini 3 Pro·high · Gemini 3 Flash·medium · Gemini 3 Flash·low · Gemini 3 Flash·minimal · Gemini 3 Flash·high · GPT-5.2·high · GPT-5.2·medium · GPT-5.2·low · Claude Haiku 4.5·nothink · GPT-5.2·nothink |
| Deictic ambiguity primary experiment balanced accuracy arxiv-2609.12162 | Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); primary experiment; read 2026-10-19 ✓ | xgen | GPT-5.2·xhigh · Gemini 3 Pro·high · Gemini 3 Flash·medium · Gemini 3 Flash·high · Gemini 3 Flash·low · Gemini 3 Flash·minimal · GPT-5.2·high · GPT-5.2·medium · GPT-5.2·low · Claude Haiku 4.5·nothink · GPT-5.2·nothink |
| FeatherBench ed-is-ai/featherbench | OpenRouter(Anthropic) · 28-task multi-domain suite (coding/data/realworld/security/tool-use), effort=high (Haiku=solo), mean cost/tokens per task-run, consolidated results/summary.json; only Opus 5.5 rung saturated ✓ | xgen | Claude Fable 5·high · Claude Fable 5.1·high · Claude Sonnet 4.6·high · Claude Sonnet 5·high · Claude Haiku 4.5·nothink · Claude Opus 5·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · GPT-5.5·high · GLM-5.2·high · GPT-5.6 Terra·high · GPT-5.6 Sol·high · GPT-5.6 Luna·high · Kimi K3·max · Gemini 3.6 Flash·high · Grok 4.5·high · Grok 4.6·high · Gemini 3.7 Flash·high · GLM-5.3·max · DeepSeek V4 Pro preview·high · DeepSeek V4.1 Flash·high · Gemini 3.8 Flash·high · GPT-6 Astra·high · GPT-6 Luna·high · GPT-6 Sol·high · Grok 4.7·high · GPT-6.1 Sol·high |
| George Liu · Opus 4.8 vs Fable 5 ai.georgeliu | Claude Code 10-prompt suite · medium/high baselines ✓ | xgen | Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Fable 5·medium · Claude Fable 5·high |
| George Liu · Sonnet 5 vs 4.6 ai.georgeliu | Claude Code 10-prompt suite · medium/high baselines ✓ | xgen | Claude Sonnet 5·medium · Claude Sonnet 5·high |
| Google model cards: DeepSWE v1.1 | DeepSWE v1.1 (Datacurve), mini-swe-agent harness; Google model-card table(s) 21 Jul 2026 (google-gemini-3.6-flash-card); 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Gemini 3.6 Flash·high · Gemini 3.5 Flash·medium |
| Google model cards: FACTS Benchmark Suite | FACTS Benchmark Suite; Google model-card table(s) Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think · GPT-5.2·xhigh |
| Google model cards: Global PIQA | Global PIQA; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | GPT-5.2·xhigh · Grok 4.1 Fast·think |
| Google model cards: GPQA Diamond (no tools) | GPQA Diamond no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | GPT-5.2·xhigh · Gemini 3.1 Pro·high |
| Google model cards: Humanity's Last Exam (full set, no tools) | HLE full set text+MM, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think · GPT-5.2·xhigh · Gemini 3.1 Pro·high |
| Google model cards: Humanity's Last Exam (search + code) | HLE full set text+MM, search (blocklist) + code execution; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | GPT-5.2·xhigh · Gemini 3.1 Pro·high |
| Google model cards: LiveCodeBench Pro (Elo) | LiveCodeBench Pro Elo; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xgen | GPT-5.2·xhigh · Gemini 3.1 Pro·high |
| Google model cards: MMMLU | MMMLU multilingual MMLU; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think · GPT-5.2·xhigh · Gemini 3.1 Pro·high |
| Google model cards: MMMU-Pro (avg Standard + Vision, no tools) | MMMU-Pro averaged over Standard (10 options) and Vision, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think · GPT-5.2·xhigh · Gemini 3.1 Pro·high |
| Google model cards: MRCR v2 8-needle 128k (average) | MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think · GPT-5.2·xhigh · Gemini 3.5 Flash-Lite·high |
| Google model cards: MRCR v2 8-needle 128k (average) | MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Gemini 3.1 Pro·high · GPT-5.2·xhigh |
| Google model cards: MRCR v2 8-needle 1M (pointwise) | MRCR v2 8-needle, 1M pointwise; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high |
| Google model cards: SimpleQA Verified | SimpleQA Verified; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Grok 4.1 Fast·think · GPT-5.2·xhigh |
| Google model cards: SWE-Bench Pro (Public) | SWE-Bench Pro public set; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Gemini 3.1 Pro·high · Gemini 3 Pro·high · GPT-5.2·xhigh · GPT-5.3-Codex·xhigh |
| Google model cards: SWE-bench Verified (single attempt) | SWE-bench Verified, single attempt; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | GPT-5.2·xhigh · Grok 4.1 Fast·think · Gemini 3.1 Pro·high |
| ianlpaterson ianlpaterson | OpenRouter · reasoning OFF ✓ | xgen | Claude Sonnet 4.6·nothink · Claude Haiku 4.5·nothink |
| MobileCybench token/cost accounting arxiv-2609.23980 | Claude Code 100-run aggregate · Opus4.8/Opus5 default, tokens+cost ✓ | xgen | Claude Opus 4.8·max · Claude Opus 5·max |
| OfficeQA arxiv-2603.08655 | Claude Agent SDK · reasoning HIGH ✓ | xgen | GPT-5.4·high · GPT-5.1·high |
| Official tau2-bench leaderboard, banking_knowledge (AllTools), Claude submissions by Sierra github-sierra-tau2-bench | banking_knowledge domain, 4 trials, user simulator gpt-5.2 (low), reasoning_effort max for all; pass^1 %; read 2026-10-18 ✓ | xgen | Claude Opus 5·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · Gemini 3.1 Pro·high · GPT-5.2·high · GPT-5.4·xhigh · GPT-5.5·xhigh · GPT-5.6 Sol·xhigh · Grok 4.5·high · Kimi K3·max · Qwen3.8 Max·xhigh |
| ponytail github-ponytail | Claude Code headless · thinking NS ✓ | xgen | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 4.8·nothink · GPT-5.5·medium |
| Qiita suwa_nobu 12 models qiita-suwanobu | Claude Code 2.1.283 · trivial read/count/write task, cold cache, median of 3, CLI cost, CLI default effort ✓ | xgen | Claude Sonnet 4.6·high · Claude Opus 4.7·xhigh · Claude Opus 4.8·high · Claude Fable 5·high · Claude Sonnet 5·high · Claude Opus 5·high · Claude Fable 5.1·high · Claude Opus 5.5·medium |
| retort exp-46/75 bookshop Clojure github-retort | Claude Code · bookshop/rest-api-crud in Clojure, Opus 5 default (n=1) vs Opus 5.5 low (n=3) ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 bookshop Rust github-retort | Claude Code · bookshop/rest-api-crud in Rust, Opus 5 default (n=1) vs Opus 5.5 low (n=3) ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil C github-retort | Claude Code · brazil-soccer-mcp in C, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil C# github-retort | Claude Code · brazil-soccer-mcp in C#, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil C++ github-retort | Claude Code · brazil-soccer-mcp in C++, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil Clojure github-retort | Claude Code · brazil-soccer-mcp in Clojure, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil Elixir github-retort | Claude Code · brazil-soccer-mcp in Elixir, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil Erlang github-retort | Claude Code · brazil-soccer-mcp in Erlang, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil Java github-retort | Claude Code · brazil-soccer-mcp in Java, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil Obj-C github-retort | Claude Code · brazil-soccer-mcp in Objective-C, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil Rust github-retort | Claude Code · brazil-soccer-mcp in Rust, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-46/75 brazil Swift github-retort | Claude Code · brazil-soccer-mcp in Swift, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Claude Opus 5·high · Claude Opus 5.5·low |
| retort exp-79 vs exp-80 python (Opus 5.5 vs Fable 5.1, isolated) github-retort | Claude Code (isolated) · rest-api-crud in python, prompt=neutral, effort=low, Opus 5.5 vs Fable 5.1, n=3 each side; requirement coverage 1.0 → cost only; read 2026-10-05 ✓ | xgen | Claude Opus 5.5·low · Claude Fable 5.1·low |
| retort exp-79 vs exp-80 rust (Opus 5.5 vs Fable 5.1, isolated) github-retort | Claude Code (isolated) · rest-api-crud in rust, prompt=neutral, effort=low, Opus 5.5 vs Fable 5.1, n=3 each side; requirement coverage 1.0 → cost only; read 2026-10-05 ✓ | xgen | Claude Opus 5.5·low · Claude Fable 5.1·low |
| SlopCode arxiv-2603.24755 | Claude Code · Reasoning HIGH ✓ | xgen | GPT-5.3-Codex·high · GPT-5.4 mini·high · GPT-5.5·high |
| SlopCodeBench Oct 2026 (humanlayer writeup): strict checkpoints passed of 196 humanlayer-scb-2026-10 | 36 problems / 196 checkpoints, one campaign per model+effort; cost = recorded estimate per campaign (USD); read 2026-10-18 ✓ | xgen | Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · GPT-6 Astra·xhigh · GPT-6 Astra·medium · GPT-6 Sol·medium · GPT-6 Sol·xhigh · GPT-6.1 Sol·medium · GPT-5.6 Sol·xhigh |
| swe-rebench swe-rebench | ReAct minimal · Opus4.8 xhigh/4.7 high, Sonnet default ✓ | xgen | GPT-5.5·xhigh · GPT-5.5·medium · Claude Opus 4.8·xhigh · GPT-5.4·medium · Claude Opus 4.7·high |
| WildClaw arxiv-2605.10912 | OpenRouter (4 harness) · thinking NS ✓ | xgen | Grok 4.20·think · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high |
| Writing-Elo effort/cost chart, Reddit 1wrkwp0 (r/LLMDevs) reddit-1wrkwp0 | same 'Writing Elo' metric family, 10 real YouTube scripts per OP, scored blind by three judges and a rubric; cost per script, log-x scatter with effort-labelled points; no board date printed on the chart; a different Reddit post/source snapshot than 1wr03jc, kept as its own board version; read 2026-10-05 ✓ | xgen | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·max · GPT-6 Luna·high · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·max · Grok 4.7·medium · Grok 4.7·high · GPT-6 Astra·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · Gemini 3.8 Flash·high |
| 1M-context retrieval: multi-hop chain retrieval (15 cells: 5 chains x 256K/512K/1M) arxiv-2605.02173 | Retrieval and multi-hop reasoning in 1M-token contexts (arXiv 2605.02173, 4 May 2026) Tables 5/6: multi-hop chain retrieval (15 cells: 5 chains x 256K/512K/1M); five models via official APIs; temperature 0 where allowed (Claude Opus 4.7 and GPT-5.5 enforce 1); score = correct cells / total as %; read 2026-10-20 ✓ | xmodel | Claude Opus 4.7·max · GPT-5.5·xhigh |
| 34bench, single Thai bus-route scan transcription, 5 runs/model (dtinth/34bench) dtinth/34bench | results.svg (table-in-SVG, parsed as text) for median accuracy (1-weighted-CER over header/forward/return/footer); meta/ | xmodel | Gemini 3.8 Flash·medium · Claude Opus 5.5·high · Claude Fable 5.1·high · Claude Sonnet 5.5·high · Gemini 3.5 Flash-Lite·minimal · Claude Haiku 5.5·nothink · Grok 4.7·high · GPT-6.1 Sol·medium · GPT-6 Sol·medium · DeepSeek V4.1 Flash·high · GPT-6 Luna·medium |
| 4DCodeBench: inverse-graphics-of-dynamic-scenes leaderboard (200 scenes), Claude Code agent 4dcodebench | arxiv.org/html/2610.03715v1 Figure 4 (src/main_leaderboard.png); Overall = mean of 5 family scores [0,1]; values printed on heatmap cells, read directly (not pixel-digitized); n=1 run/model, no repeats stated; read-from-image; read 2026-10-05 ✓ | xmodel | GPT-6 Astra·max · Claude Opus 5.5·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-5.6 Luna·max · GPT-5.6 Sol·high · Gemini 3.8 Flash·high · GPT-5.6 Terra·high |
| AA /task v3 artificialanalysis | AA-Index v3 · max ✓ | xmodel | DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high · GLM-4.7·think |
| AA /task v4 artificialanalysis | AA launch article · per-task $, Fable 5.1 xhigh/max vs Fable 5/Opus 5 max ✓ | xmodel | Gemini 3.5 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · GLM-4.7·think · GPT-5.1·high · GPT-5.4·xhigh · GPT-5.5·low · GPT-5.5·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.8 Max·xhigh |
| AA AutomationB. artificialanalysis | AA evaluation page · 657 tasks partial-credit, 3 models at max ✓ | xmodel | Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GLM-5.3·low · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·nothink · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3-Coder-Next·nothink · Gemini 3.8 Flash·low · Gemini 3.1 Flash-Lite·high · GPT-5.6 Sol·nothink · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · Claude Opus 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Haiku 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5·max · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Opus 4.8·max · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·max · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| AA Briefcase v1.1 (xAI Grok 4.7 post) xai | AA Briefcase v1.1 Elo, multi-hour office work; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · GPT-5.6 Sol·max · Claude Fable 5.1·max · GPT-6 Astra·max |
| AA FrontierCode 1.1 (AA-run, weighted score) artificialanalysis | Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Luna·medium · GPT-6.1 Sol·high · GPT-6 Luna·nothink · Gemini 3.8 Flash·medium · GPT-6 Luna·high · GPT-6.1 Sol·medium · GPT-6 Luna·low · GLM-5.3·low · GPT-6.1 Sol·low · Gemini 3.8 Flash·low · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · GLM-5.3·max · GLM-5.3-Flash·max · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·low · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · DeepSeek V4 Pro 0813 build·max |
| AA MMMU-Pro artificialanalysis | Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | GPT-6 Luna·nothink · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · GPT-5.1·high · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.4·low · GPT-5.4 mini·xhigh · GPT-5.4 mini·medium · GPT-5.4 mini·nothink · GPT-5.4 nano·medium · GPT-5.4·nothink · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·nothink · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·max · GPT-6.1 Sol·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·xhigh · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.20·think · Grok 4.20·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.7·low · Kimi K2.6·think · Kimi K3·max · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Claude Opus 5·xhigh · Claude Opus 5·low · Grok 4.1 Fast·think · Claude Sonnet 5·max · Grok 4.1 Fast·nothink · Gemini 3.1 Flash-Lite·high · Claude Opus 5·high · Claude Opus 5·max · Claude Haiku 4.5·nothink · Claude Opus 4.7·max · Claude Opus 5·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · GPT-5.4 nano·xhigh · GPT-5.4 nano·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink |
| AA-AnalystAgent artificialanalysis | Artificial Analysis AA-AnalystAgent; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash·max · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · GLM-5.3·max · GLM-5.3-Flash·max · GPT-5.4 mini·xhigh · GPT-5.5·xhigh · GPT-5.5·nothink · GPT-5.6 Sol·max · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Grok 4.7·high · Kimi K3·max · Claude Sonnet 5·max · Gemini 3.1 Flash-Lite·high · Claude Opus 4.8·max · Claude Opus 5·max · Claude Fable 5·max · Claude Opus 4.7·max · Gemini 3.5 Flash·high |
| AA-Briefcase (xAI Grok 4.6 post) xai | AA-Briefcase Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · GPT-5.6 Sol·max · Claude Fable 5·max |
| AA-Briefcase Elo, Artificial Analysis article of 21 July 2026 artificialanalysis-article | AA-Briefcase Elo as printed in the AA article "Kimi K3: second only to Fable 5 on AA-Briefcase" (2026-07-21); earlier snapshot than the live board; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·max · Claude Sonnet 5·max · Claude Opus 4.8·max |
| aa-index411 tabbit.ai | — | xmodel | Claude Opus 5·max · GLM-5.3·max · Kimi K3·max · GPT-5.6 Sol·max · DeepSeek V4 Pro 0813 build·max · GPT-5.6 Luna·max · Gemini 3.7 Flash·high · Grok 4.6·high |
| aa-index5 artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · Claude Opus 5.5·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·medium · Claude Opus 5.5·high · Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Hydrogen·low · Gemini 4 Hydrogen·medium · Gemini 4 Hydrogen·high · Gemini 4 Hydrogen·xhigh · GPT-6.1 Terra·low · GPT-6.1 Terra·medium · GPT-6.1 Terra·high · GPT-6.1 Terra·xhigh · GPT-6.1 Terra·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·nothink · GLM-5.3·low · GLM-5.3·max · Claude Fable 5.2·max · Claude Fable 5.2·xhigh · Claude Fable 5.2·high · Claude Fable 5.2·medium · Claude Fable 5.2·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| aa5-aa-briefcase artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·low · GPT-6 Luna·nothink · Qwen3.8-27B·medium · Gemini 3.8 Flash·medium · GPT-6 Luna·high · Qwen3.8-27B·low · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · GPT-6 Astra·max · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Haiku 5.5·max · GPT-6.1 Terra·max · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Gemini 4 Argon·high · Claude Opus 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Haiku 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| aa5-code2video-aa artificialanalysis | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Haiku 5.5·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GLM-5.3·max · Gemini 4 Argon·high · Claude Fable 5.1·max |
| aa5-critpt-v2-hard artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Haiku 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| aa5-frontiercode-2-0 artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Haiku 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| aa5-harvey-lab-aa artificialanalysis | — | xmodel | Claude Haiku 5.5·high · GPT-6 Astra·max · Claude Opus 5.5·max · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GLM-5.3·max · Gemini 4 Argon·high · Claude Fable 5.1·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max |
| aa5-healthbench artificialanalysis | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Haiku 5.5·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GLM-5.3·max · Gemini 4 Argon·high · Claude Fable 5.1·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max |
| aa5-healthbench-professional artificialanalysis | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Haiku 5.5·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GLM-5.3·max · Gemini 4 Argon·high · Claude Fable 5.1·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max |
| aa5-hle-diamond artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Haiku 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| aa5-mka artificialanalysis | — | xmodel | Gemini 4 Carbon·high · Gemini 4 Argon·high · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · GPT-6.1 Terra·max · DeepSeek V4.1 Flash 1017 build·max |
| aa5-mlcr-aa artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · Claude Opus 5.5·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·high · Claude Fable 5.2·max |
| aa5-omniscience-non-hallucination artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · Claude Opus 5.5·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max |
| aa5-vials-aa artificialanalysis | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Haiku 5.5·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.2·max |
| aaapex artificialanalysis | — | xmodel | Kimi K3·max · Gemini 3.1 Flash-Lite·high · GLM-5·think · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·max · Grok 4.20·think · Grok 4.3·high · Kimi K2.6·think · Gemini 3.5 Flash·high · GPT-5.4 nano·xhigh |
| aaenterpriseops artificialanalysis | — | xmodel | Claude Fable 5·max · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.7 Flash·medium · GPT-5.4 mini·xhigh · GPT-5.5·medium · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K2.6·think · Kimi K2.7 Code·solo · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · Claude Sonnet 5·xhigh · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · GPT-5.4 nano·xhigh |
| aagpqa artificialanalysis | — | xmodel | Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · GLM-5.3·max · GLM-5.3-Flash·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·max · GPT-6 Astra·medium · GPT-6 Astra·xhigh · Kimi K3·max · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Claude Fable 5·max · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · GLM-4.7·think · GLM-5·think · GLM-5.1·think · GPT-5.1·high · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.2·xhigh · GPT-5.3-Codex·xhigh · GPT-5.4·low · GPT-5.4·nothink · GPT-5.4·xhigh · GPT-5.4 mini·medium · GPT-5.4 mini·nothink · GPT-5.4 mini·xhigh · GPT-5.4 nano·medium · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.5·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·max · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·nothink · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Claude Haiku 4.5·nothink · Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Qwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Claude Sonnet 5·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · GPT-5.4 nano·xhigh · GPT-5.4 nano·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Grok 4.20·think · Qwen3.8 Max·xhigh |
| aaifbench artificialanalysis | — | xmodel | Claude Fable 5·max · Gemini 3.1 Flash-Lite·high · GLM-4.7·think · GLM-5·think · GLM-5.1·think · GPT-5.1·high · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.2·xhigh · GPT-5.3-Codex·xhigh · GPT-5.4·low · GPT-5.4·nothink · GPT-5.4·xhigh · GPT-5.4 mini·medium · GPT-5.4 mini·nothink · GPT-5.4 mini·xhigh · GPT-5.4 nano·medium · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.5·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Claude Haiku 4.5·nothink · Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Claude Opus 4.7·max · Claude Opus 4.8·max · Qwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · GPT-5.4 nano·xhigh · GPT-5.4 nano·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink |
| aaime25 artificialanalysis | — | xmodel | GLM-4.7·think · GPT-5.1·high · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.2·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Claude Haiku 4.5·nothink · Gemini 3 Pro·high |
| aalcb artificialanalysis | — | xmodel | GLM-4.7·think · GPT-5.1·high · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.2·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Claude Haiku 4.5·nothink · Gemini 3 Pro·high |
| aaomni-index artificialanalysis | — | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·xhigh · GPT-6 Luna·xhigh · GPT-6 Astra·xhigh · Claude Haiku 5.5·high · Grok 4.7·low · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Astra·max · GPT-6 Luna·medium · GPT-6.1 Sol·high · Claude Haiku 5.5·low · Grok 4.7·xhigh · GPT-6 Luna·nothink · Qwen3.8-27B·medium · GPT-6 Luna·max · Gemini 3.8 Flash·medium · Claude Haiku 5.5·max · GPT-6 Luna·high · Qwen3.8-27B·low · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Sonnet 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·low · GPT-6 Luna·low · GLM-5.3·low · GLM-5.3·max · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6.1 Sol·low · GPT-6 Astra·medium · Gemini 3.8 Flash·low · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Haiku 5.5·xhigh · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3 Pro·high · GLM-4.7·think · GLM-5·think · GLM-5.1·think · GLM-5.3-Flash·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.4·low · GPT-5.4 mini·xhigh · GPT-5.4 mini·medium · GPT-5.4 mini·nothink · GPT-5.4 nano·xhigh · GPT-5.4 nano·medium · GPT-5.4 nano·nothink · GPT-5.4·nothink · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.6 Luna·max · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·max · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·nothink · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · GPT-6 Sol·max · GPT-6 Sol·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Sol·xhigh · Grok 4.20·think · Grok 4.20·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Kimi K3·max · Qwen3.5-122B-A10B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.8-27B·xhigh · Qwen3-Coder-Next·nothink · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Gemini 3.1 Flash-Lite·high · Claude Haiku 4.5·nothink · Claude Opus 4.7·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5·high · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| aatau-banking artificialanalysis | — | xmodel | Claude Fable 5·max · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · GLM-4.7·think · GLM-5.1·think · GPT-5.1·high · GPT-5.2·nothink · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.5·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·max · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·nothink · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Kimi K2.6·think · Kimi K2.7 Code·solo · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Qwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Claude Sonnet 5·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · GPT-5.4 nano·xhigh · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Claude Fable 5.1·low · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · GLM-5.3·max · GLM-5.3-Flash·max · Kimi K3·max · Qwen3.8-27B·xhigh · Qwen3.8-27B·low · Qwen3.8-27B·medium · Claude Fable 5.1·medium · Claude Fable 5.1·high · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · GPT-6 Astra·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh |
| aatau2 artificialanalysis | — | xmodel | Claude Fable 5·max · Gemini 3.1 Flash-Lite·high · GLM-4.7·think · GLM-5·think · GLM-5.1·think · GPT-5.1·high · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.2·xhigh · GPT-5.3-Codex·xhigh · GPT-5.4·low · GPT-5.4·nothink · GPT-5.4·xhigh · GPT-5.4 mini·medium · GPT-5.4 mini·nothink · GPT-5.4 mini·xhigh · GPT-5.4 nano·medium · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.5·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Claude Haiku 4.5·nothink · Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Claude Opus 4.7·max · Claude Opus 4.8·max · Qwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · GPT-5.4 nano·xhigh · GPT-5.4 nano·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink |
| aatb21 artificialanalysis | — | xmodel | Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · GLM-5.3·max · GLM-5.3-Flash·max · GPT-6 Astra·high · GPT-6 Astra·low · GPT-6 Astra·max · GPT-6 Astra·medium · GPT-6 Astra·xhigh · Kimi K3·max · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Claude Fable 5·max · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · GLM-4.7·think · GLM-5.1·think · GPT-5.1·high · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.5·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·low · GPT-5.6 Luna·max · GPT-5.6 Luna·medium · GPT-5.6 Luna·nothink · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·nothink · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·nothink · GPT-5.6 Terra·xhigh · Grok 4.3·high · Grok 4.3·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Kimi K2.6·think · Kimi K2.7 Code·solo · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Qwen3-Coder-Next·nothink · Qwen3.6-27B·nothink · Claude Sonnet 5·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · GPT-5.4 nano·xhigh · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.8 Max·xhigh |
| aatbhard artificialanalysis | — | xmodel | Claude Fable 5·max · Gemini 3.1 Flash-Lite·high · GLM-4.7·think · GLM-5·think · GLM-5.1·think · GPT-5.1·high · GPT-5.2·medium · GPT-5.2·nothink · GPT-5.2·xhigh · GPT-5.3-Codex·xhigh · GPT-5.4·low · GPT-5.4·nothink · GPT-5.4·xhigh · GPT-5.4 mini·medium · GPT-5.4 mini·nothink · GPT-5.4 mini·xhigh · GPT-5.4 nano·medium · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.5·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·xhigh · Grok 4.1 Fast·think · Grok 4.1 Fast·nothink · Grok 4.3·high · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·nothink · Claude Haiku 4.5·nothink · Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Claude Opus 4.7·max · Claude Opus 4.8·max · Qwen3-Coder-Next·nothink · Qwen3.5-27B·nothink · Qwen3.6-27B·nothink · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3 Pro·high · GPT-5.4 nano·xhigh · GPT-5.4 nano·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink |
| adebench github-ade-bench | — | xmodel | Claude Fable 5·low · Claude Opus 4.8·xhigh |
| Agent Island social-deduction arena (Plackett-Luce skill), OpenRouter models arxiv-2605.04312 | Agent Island (arXiv 2605.04312, 5 May 2026): 999 multi-agent social-deduction games with 49 models via OpenRouter, Bayesian Plackett-Luce posterior skill (mean of lambda; prior mean 1; Table 1); reasoning effort not stated; no cost; read 2026-10-20 ✓ | xmodel | Kimi K2 Thinking·solo · Grok 4.20·think |
| AgentHop agentic multi-hop scientific QA accuracy and cost agenthop-2609.34428 | AgentHop 1,011 four-option items, 7 tools, 20 turns, 200K-token cap, one attempt, same harness for all; metric accuracy 0-1; cost = USD for the full 1,011-item pass at official per-token pricing recorded in the released harness (price date not given); arXiv 2609.34428 Tables 2/8 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.3-Codex·medium · GPT-5.4·medium |
| Agents on Guren, Stage 2 re-run, bare condition (gurenjs/agents-on-guren) gurenjs/agents-on-guren | RESULTS-STAGE2.md at commit 2b43507 ("Stage 2 re-run with auto memory off and Sonnet 5.5", 2026-09-30), bare condition; Claude Code 2.1.284, 3 trials per cell, 9 tasks; pass is hidden-behaviour-test pass/fail per cell (27 cells); cost is median API-equivalent USD per cell from Claude Code total_cost_usd; effort: run.sh passes only --model "$MODEL", no effort flag, so default for every model; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·high |
| Agents on Guren, Stage 2 re-run, shipped condition (gurenjs/agents-on-guren) gurenjs/agents-on-guren | RESULTS-STAGE2.md at commit 2b43507 ("Stage 2 re-run with auto memory off and Sonnet 5.5", 2026-09-30), shipped condition; Claude Code 2.1.284, 3 trials per cell, 9 tasks; pass is hidden-behaviour-test pass/fail per cell (27 cells); cost is median API-equivalent USD per cell from Claude Code total_cost_usd; effort: run.sh passes only --model "$MODEL", no effort flag, so default for every model; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·high |
| Agents on Rails · atomic rubyonrails | rubyonrails.org/ai + rails/ai-evals runs · 21 Writebook tasks ×3, miniswen, list-price cost/run ✓ | xmodel | Kimi K3·max · GPT-5.6 Sol·medium · Claude Opus 5·high · Claude Fable 5.1·high · Claude Fable 5·high · Claude Opus 4.8·high · Claude Sonnet 5·high |
| Agents' Last Exam (OpenAI GPT-5.6 post chart) openai | Agents' Last Exam, professional workflows, code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max |
| Agents' Last Exam (OpenAI GPT-6 Astra post chart) openai | Agents' Last Exam; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Claude Fable 5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| Agents' Last Exam (OpenAI GPT-6 Sol and Luna post) openai | Agents' Last Exam; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Claude Fable 5·xhigh · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| Agents' Last Exam ALE-CLI (GLM-5.3 card) zhipu | ALE official protocol, Claude Code harness, 105 tasks, effort=max 1M context 64K output, 4h timeout; GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Qwen3.8 Max·xhigh · Claude Opus 4.8·max · Claude Fable 5·max |
| AgentTime (arXiv 2610.09944) Agents’ Last Exam under duration requests arxiv-2610.09944 | AgentTime Table 5 column Agents’ Last Exam: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) AppWorld under duration requests arxiv-2610.09944 | AgentTime Table 5 column AppWorld: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) AssistantBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column AssistantBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) CORE-Bench v1.1 under duration requests arxiv-2610.09944 | AgentTime Table 5 column CORE-Bench v1.1: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) DeepSWE v1.1 under duration requests arxiv-2610.09944 | AgentTime Table 5 column DeepSWE v1.1: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) GPQA Diamond under duration requests arxiv-2610.09944 | AgentTime Table 5 column GPQA Diamond: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) Humanity’s Last Exam under duration requests arxiv-2610.09944 | AgentTime Table 5 column Humanity’s Last Exam: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) METR public tasks under duration requests arxiv-2610.09944 | AgentTime Table 5 column METR public tasks: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) OSWorld 2.0 under duration requests arxiv-2610.09944 | AgentTime Table 5 column OSWorld 2.0: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) PaperBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column PaperBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) PostTrainBench v1.1 under duration requests arxiv-2610.09944 | AgentTime Table 5 column PostTrainBench v1.1: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) PPTArena under duration requests arxiv-2610.09944 | AgentTime Table 5 column PPTArena: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) ProgramBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column ProgramBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) Sakana ALE-Bench under duration requests arxiv-2610.09944 | AgentTime Table 5 column Sakana ALE-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) Terminal-Bench 4.0 under duration requests arxiv-2610.09944 | AgentTime Table 5 column Terminal-Bench 4.0: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) TUA-Bench under duration requests arxiv-2610.09944 | AgentTime Table 5 column TUA-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) WildClawBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column WildClawBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AgentTime (arXiv 2610.09944) YC-Bench under duration requests arxiv-2610.09944 | AgentTime Table 5 column YC-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GPT-6 Astra·max |
| AI Coding Daily LLM Coding Leaderboard (4 projects, max 70) aicodingdaily | Laravel 20 + React-TS 20 + bug finding 10 + edge-case 20 = 70; Claude Code rows; author-scored; page 'Last update: October 17th, 2026'; read 2026-10-18 ✓ | xmodel | GPT-6.1 Sol·high · Claude Opus 5.5·high · GPT-6.1 Sol·medium · GPT-6 Astra·high · GPT-6 Astra·medium · Claude Opus 5.5·medium · GPT-6.1 Sol·low · Claude Sonnet 5.5·high · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.1·medium · Claude Opus 5.5·low · Claude Sonnet 5.5·medium · Grok 4.7·high · GPT-6 Sol·high · GPT-6 Luna·max · Claude Haiku 5.5·high · Claude Haiku 5.5·medium · Gemini 4 Argon·high · Grok 4.6·high · GPT-6 Luna·xhigh · GPT-6 Sol·medium · GPT-6 Luna·high · Gemini 3.8 Flash·high · GPT-6 Luna·medium |
| AI for Mortals · Backrooms game aiformortals | Claude Code /goal, high effort, Blender + web game, one build per model, cost only, 29 Sep ✓ | xmodel | Claude Sonnet 5·high · Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AI for Mortals · demoscene aiformortals | Claude Code /goal, high effort, code-only demo, one build per model, cost only, earlier round ✓ | xmodel | Claude Sonnet 5·high · Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AI for Mortals · launch video aiformortals | Claude Code /goal, high effort, one build per model, no numeric score (cost, tokens, time only), 29 Sep ✓ | xmodel | Claude Sonnet 5·high · Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AI for Mortals · RPG aiformortals | Claude Code /goal, high effort, Diablo-style RPG, one build per model (no Sonnet 5), cost only, earlier round ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AI for Mortals · website clone aiformortals | Claude Code /goal, high effort, prompt as file, one build per model, cost only, 29 Sep ✓ | xmodel | Claude Sonnet 5·high · Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high) aibenchy | aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·low · Claude Opus 5·high · Claude Opus 5·medium · Claude Opus 5.5·medium · Claude Sonnet 5.5·max · Claude Opus 5·low · Claude Fable 5·medium · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Sonnet 5·medium · Claude Haiku 5.5·max · Claude Haiku 5.5·high · Claude Opus 4.8·medium · Claude Opus 4.7·medium · Claude Haiku 5.5·medium · Claude Opus 4.8·low · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · GPT-6 Sol·high · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6.1 Sol·xhigh · GPT-6 Astra·high · GPT-6 Astra·xhigh · Gemini 3.7 Flash·high · Gemini 3.6 Flash·medium · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-5.6 Sol·low · GPT-6 Astra·low · Gemini 3 Flash·medium · GPT-6.1 Sol·low · Gemini 3.7 Flash·medium · GPT-6 Sol·medium · GPT-5.6 Sol·medium · GPT-6 Astra·medium · GPT-5.6 Sol·high · GPT-5.5·low · Gemini 3.7 Flash·low · GPT-5.5·medium · GPT-5.3-Codex·medium · Qwen3.8 Max 0902 build·medium · Gemini 3.5 Flash-Lite·high · Gemini 4 Argon·medium · Gemini 3.6 Flash·low · GLM-5.3-Flash·max · Qwen3.8 Max·low · GPT-5.4·medium · GPT-5.2·medium · GPT-6 Sol·low · Qwen3.8 Max 0902 build·low · Grok 4.6·high · Gemini 3.1 Pro·medium · DeepSeek V4 Flash 0731 build·high · GLM-5.2·high · GPT-5.6 Terra·high · Gemini 3.5 Flash·high · GPT-6 Luna·medium · Grok 4.5·high · GPT-5.6 Terra·medium · DeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high · GPT-5.6 Luna·high · GPT-5.4 nano·medium · Gemini 3.8 Flash·medium · GPT-5.6 Terra·low · GLM-5.3-FlashX·max · GPT-5.4 mini·medium · Gemini 3.5 Flash·medium · Grok 4.5·medium · GLM-5.3·high · Grok 4.6·medium · Grok 4.5·low · GPT-6 Luna·high · GPT-5.6 Luna·medium · Grok 4.7·low · GLM-5.3-Flash·high · Grok 4.7·medium · Gemini 3.5 Flash-Lite·medium · Qwen3.5-27B·nothink · Gemini 3.8 Flash·low · Grok 4.6·low · Gemini 3 Flash·low · Gemini 3.1 Flash-Lite·medium · GPT-5.6 Luna·low · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GLM-5.3-Flash·low · GPT-5.5·nothink · GLM-5.3·low · Grok 4.3·medium · Qwen3.5-122B-A10B·nothink · Gemini 3.5 Flash·minimal · Qwen3.6-27B·nothink · Gemini 3.5 Flash-Lite·low · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·minimal · Qwen3.7 Flash·nothink · GPT-5.6 Terra·nothink · GPT-5.4 mini·nothink · GPT-5.4·nothink · Qwen3.5-35B-A3B·nothink · Kimi K2.6·nothink · GPT-5.6 Luna·nothink · Qwen3-Coder-Next·nothink · GPT-5.4 nano·nothink |
| AIFM: Adizero shoe in Blender (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AIFM: Awwwards clone (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AIFM: Launch film (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AIFM: Mana Yerba Mate site (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AIFM: Moby Dick illustration in code (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AIFM: RollerCoaster Tycoon clone (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AIFM: THPS-style 3D game (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| AIME 2025 (GLM-4.7 card) zhipu | AIME 2025; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| aimecontract arxiv-2608.16956 | — | xmodel | Claude Sonnet 5·high · Claude Fable 5·high · GPT-5.6 Terra·medium · GPT-5.4 mini·low · GPT-5.4 mini·high |
| AIMultiple text-to-SQL (Claude Code) aimultiple.com | AIMultiple AIM-Text-to-SQL, Opus 5.5 and Fable 5.1 in Claude Code at medium effort · SQL accuracy in %; each model answered its own subset of questions (506 and 491); no cost; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·medium · GPT-6 Astra·medium |
| AIReiter aireiter | Claude Code · high ✓ | xmodel | Claude Sonnet 5·xhigh · Claude Opus 4.8·medium · Claude Sonnet 5·high · Claude Opus 4.8·high |
| AI生活 (YouTube): 25 questions x 99 runs on claude.ai chat, Haiku 5.5 vs Sonnet 5.5 vs Opus 5.5 youtube-aiseikatsu | claude.ai chat (not API); new chat per question; Q1-12 x5 + Q13-25 x3 = 99 runs; same grading as 29 Sep; runs on different days (Opus 26+29 Sep; Sonnet 29 Sep; Haiku 8 Oct); read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·high · Claude Haiku 5.5·max · Claude Sonnet 5.5·high · Claude Opus 5.5·high |
| ALE-Bench epoch-archive | Epoch archive · 3 models at matched high, measured token splits ✓ | xmodel | Claude Opus 5.5·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-5.1-Codex-Max·high · Grok 4.20·think · GPT-6 Astra·max · GPT-6 Sol·max · GPT-5.6 Sol·max · Claude Opus 5·high · Claude Fable 5.1·high · Claude Fable 5·high · GPT-5.6 Terra·max · GPT-5.5·xhigh · Claude Sonnet 5.5·high · GPT-5.6 Luna·max · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.5·medium · GPT-6 Luna·xhigh · Claude Opus 4.8·high · Kimi K3·max · GPT-5.4·medium · Grok 4.6·xhigh · Claude Sonnet 5·high · DeepSeek V4 Pro 0813 build·max · GLM-5.3·high · Grok 4.5·high · GPT-5.2·high · Gemini 3.8 Flash·high · GPT-5.2·medium · GPT-5.1·high · GPT-5.4 mini·high · GPT-5.5·nothink · DeepSeek V4.1 Flash·max · GPT-5.4·nothink · GLM-5.2·high · GPT-5.4 nano·high · Gemini 3.5 Flash·high · Gemini 3.7 Flash·high · Kimi K2.7 Code·solo · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · GLM-5.3-Flash·high |
| AlgoWorlds (240 algorithmic worlds), exact optimality arxiv-2608.29397 | AlgoWorlds (arXiv 2608.29397, 29 Aug 2026): 240 algorithmic worlds, 3 trials, agent acquires information with paid tools then decides; score = exact optimality % (first final-decision metric of Table 2; feasibility, reference utility, sufficiency, coverage are other columns of the same runs); acquisition spend in tool-budget units, not USD (no cost row); read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.6 Sol·max · Claude Sonnet 5·max · GPT-5.6 Terra·max |
| AminBlg/SimpleEnglish: ASD-STE100 skill vs baseline, 9 Claude models x 8 tasks, violations per 100 words (regex linter), --effort low github-simpleenglish | evals/results/RESULTS.md (pushed 5 Oct); score = reduction of STE violations per 100 words with the skill vs baseline (higher is better); output tokens = skill condition; read 2026-10-18 ✓ | xmodel | Claude Fable 5.1·low · Claude Fable 5·low · Claude Opus 5·low · Claude Opus 4.8·low · Claude Sonnet 5·low |
| Analog Design Bench: pass@1 arxiv-2609.33356 | Analog Design Bench (arXiv 2609.33356 v1 27 Sep 2026); 50 transistor-level analog design tasks x 3 rollouts = 150 two-hour attempts per configuration; main metric pass@1 (full-spec pass); GPT/Claude in native Codex/Claude Code, others via Claude Code on compatible APIs; cost = per-attempt mean; read 2026-10-20 ✓ | xmodel | GPT-5.6 Luna·max · GPT-5.6 Terra·max · GPT-5.5·xhigh · GPT-5.6 Sol·max · Claude Opus 5·max · Claude Fable 5·max |
| android-bench-2 | — | xmodel | Claude Sonnet 5.5·max · Claude Opus 5.5·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · GPT-6 Astra·high · Claude Fable 5.1·high · Gemini 4 Argon·high · Kimi K3·max |
| anthropic-cookbook-latency-desk-agent anthropic | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low · Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·high · Claude Haiku 5.5·medium · Claude Haiku 5.5·low |
| anthropic-cookbook-latency-desk-agent-fast-mode anthropic | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low |
| anthropic-cookbook-latency-desk-agent-race anthropic | — | xmodel | Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·high · Claude Haiku 5.5·medium |
| anthropic-cookbook-latency-desk-agent-race-bigger-pages anthropic | — | xmodel | Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·high · Claude Haiku 5.5·medium |
| anthropic-cookbook-latency-desk-agent-race-fast anthropic | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low |
| anthropic-cookbook-latency-desk-agent-race-fast-bigger-pages anthropic | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low |
| anthropic-cookbook-latency-desk-agent-race-fast-prefetch anthropic | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low |
| anthropic-cookbook-latency-desk-agent-race-fast-prefetch-bigger-pages anthropic | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low |
| anthropic-cookbook-latency-desk-agent-race-prefetch anthropic | — | xmodel | Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·high · Claude Haiku 5.5·medium |
| anthropic-cookbook-latency-desk-agent-race-prefetch-bigger-pages anthropic | — | xmodel | Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·high · Claude Haiku 5.5·medium |
| APEX-Agents (xAI Grok 4.6 post) xai | APEX-Agents %, harness not stated (Mercor's benchmark by name; page does not attribute); xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · GPT-5.6 Sol·max · Claude Fable 5·max |
| APEX-Agents public leaderboard rows quoted by StagedWorkspace (Pass@1) stagedws-2608.18050 | APEX-Agents public leaderboard rows quoted as published by arXiv 2608.18050 Table 2b (Pass@1 with 8 independent attempts per task), snapshot unknown (older than the held Mercor board: cells differ); same-benchmark-as:mercor-apexagents; metric Pass@1 %; read 2026-10-20 ✓ | xmodel | Gemini 3 Flash·high · GPT-5.4 nano·xhigh · GPT-5.4 mini·xhigh · GPT-5.4·xhigh |
| APEX-Harm mean task score under attack (all 6 categories) apexharm-2610.13623 | APEX-Harm 180 attack task instances per model (15 underlying APEX-Agents tasks x 12 attack variants), APEX-Agents harness with MCP tools, 150 steps, no timer, judge/rules per paper; metric mean task score 0-1 (utility under attack); ASR/ER safety rates not extracted; arXiv 2610.13623 Table 4 (Oct 2026); cost = Table 1 total for Gemini 3.8 Flash only; read 2026-10-20 ✓ | xmodel | Claude Opus 5·max · Claude Sonnet 5·max · GPT-6 Astra·low · Gemini 3.8 Flash·high · GPT-5.6 Luna·low · Gemini 3.7 Flash·high · Kimi K3·max · GPT-5.6 Sol·high · GLM-5.3-Flash·max · GPT-5.6 Terra·high · Gemini 3.6 Flash·high |
| apexharm-asr apexharm-2610.13623 | — | xmodel | Claude Opus 5·max · Claude Sonnet 5·max · GPT-6 Astra·low · Gemini 3.8 Flash·high · GPT-5.6 Luna·low · Gemini 3.7 Flash·high · Kimi K3·max · GPT-5.6 Sol·high · GLM-5.3-Flash·max · GPT-5.6 Terra·high · Gemini 3.6 Flash·high |
| ARC-AGI-1 (Verified) (table) openai | ARC-AGI-1 (Verified); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| ARC-AGI-2 (Verified) (table) openai | ARC-AGI-2 (Verified); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| ARC-AGI-3 (semi-private) arcprize | ARC Prize ARC-AGI-3 semi-private interactive-game set, score in % (as published x100), cost = total run USD; Haiku 5.5 on the Standard harness (the Provider-Adapter harness entry, 2.82 % / $437.14, is a different configuration and is not mixed in); read 2026-10-15 ✓ | xmodel | GPT-5.4·high · GPT-5.5·high · Claude Opus 4.7·high · Claude Opus 4.8·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · Grok 4.5·high · Grok 4.5·medium · Grok 4.5·low · Claude Opus 5·high · Grok 4.6·xhigh · GPT-6 Astra·max · GPT-6 Astra·xhigh · GPT-6 Astra·high · GPT-6 Astra·medium · GPT-6 Astra·low · GPT-6 Luna·max · GPT-6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Luna·medium · GPT-6 Luna·low · GPT-6 Luna·nothink · Gemini 3.8 Flash·high · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·low · GPT-6 Sol·max · GPT-6 Sol·xhigh · GPT-6 Sol·high · GPT-6 Sol·medium · GPT-6 Sol·low · GPT-6 Sol·nothink · GPT-6.1 Sol·max · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·high · GPT-6.1 Sol·medium · GPT-6.1 Sol·low · Grok 4.7·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·low · Claude Haiku 5.5·max · Gemini 4 Argon·medium · Gemini 4 Argon·low · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.2·xhigh · Claude Fable 5.2·high · Claude Fable 5.2·medium · Claude Fable 5.2·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| arcagi3-provider-adapter arcprize | — | xmodel | Claude Fable 5.2·max · Claude Fable 5.2·xhigh · Claude Fable 5.2·high · Claude Fable 5.2·medium · Claude Fable 5.2·low · Gemini 4 Carbon·high · Gemini 4 Carbon·medium · Gemini 4 Carbon·low |
| Argo-Bench: enterprise data-agent tasks (210 tasks), Inspect AI 0.3.263 harness argo-bench | arxiv.org/html/2610.02122v1 Table 2 (best-effort snapshot, extra-high where offered) + Table 10 (effort ladder, mean score /100, tasks completed at every level); n=1 rollout/task; score = mean task score out of 100 with a model's forecast tasks floored at 0 as a block; cost = API spend per task, excluding BigQuery (Table 2 only, not itemized by effort in Table 10); read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high |
| ArtifactArena model leaderboard, Design Lab Harness (DLH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Fable 5·high · Claude Sonnet 5·high · Claude Opus 4.8·high · GPT-6 Astra·high · Gemini 3.8 Flash·high · GPT-5.4·high · GPT-5.3-Codex·high · GPT-5.6 Sol·high · GPT-5.6 Terra·high · Gemini 3.1 Pro·high · GLM-5.2·high · GPT-5.6 Luna·high · GLM-5.3·high · GPT-5.5·high |
| ArtifactArena model leaderboard, Sampling Harness (SH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Fable 5·high · Claude Sonnet 5·high · Claude Opus 4.8·high · GPT-6 Astra·high · Gemini 3.8 Flash·high · GPT-5.4·high · GPT-5.3-Codex·high · GPT-5.6 Sol·high · Grok 4.6·high · GPT-5.6 Terra·high · Gemini 3.1 Pro·high · GLM-5.2·high · GPT-5.6 Luna·high · GPT-5.5·high · Grok 4.5·high |
| ArtifactArena model leaderboard, Verifier-Grounded Refinement Harness (VGH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Fable 5·high · Claude Sonnet 5·high · Claude Opus 4.8·high · GPT-6 Astra·high · Gemini 3.8 Flash·high · GPT-5.4·high · GPT-5.3-Codex·high · GPT-5.6 Sol·high · Grok 4.6·high · GPT-5.6 Terra·high · Gemini 3.1 Pro·high · GLM-5.2·high · GPT-5.6 Luna·high · GLM-5.3·high · GPT-5.5·high · Grok 4.5·high |
| Artificial Analysis Coding Agent Index v1.1 (OpenAI GPT-5.6 post chart) openai | Artificial Analysis Coding Agent Index v1.1; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Claude Opus 4.8·medium · Claude Opus 4.8·max |
| Artificial Analysis Coding Agent Index v1.4 (OpenAI GPT-6 Astra post chart, code) openai | Artificial Analysis Coding Agent Index v1.4; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| Artificial Analysis Coding Agent Index v1.4 (OpenAI GPT-6 Astra post chart, non-code) openai | Artificial Analysis Coding Agent Index v1.4; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Gemini 3.8 Flash·high · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| Artificial Analysis Intelligence Index (OpenAI GPT-5.5 post chart) openai | Artificial Analysis Intelligence Index; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.5·high · Claude Opus 4.7·max · GPT-5.4·xhigh · GPT-5.5·medium · GPT-5.5·low · GPT-5.4·low · GPT-5.5·nothink · GPT-5.4·nothink |
| arxiv-2604.26954-kappa arxiv-2604.26954 | — | xmodel | GPT-5.4 nano·nothink · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·high · GPT-5.4 mini·nothink · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·high · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · Gemini 3 Flash·minimal · Gemini 3 Flash·low · Gemini 3 Flash·medium · Gemini 3 Flash·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high |
| arxiv-2604.26954-kappa-temp0-j1 arxiv-2604.26954 | — | xmodel | GPT-5.4 nano·nothink · GPT-5.4 mini·nothink · GPT-5.4·nothink · Gemini 3 Flash·minimal |
| arxiv-2604.26954-kappa-temp1-j3 arxiv-2604.26954 | — | xmodel | GPT-5.4 nano·nothink · GPT-5.4 mini·nothink · GPT-5.4·nothink · Gemini 3 Flash·minimal |
| arxiv-2604.26954-kappa-temp1-j5 arxiv-2604.26954 | — | xmodel | GPT-5.4 nano·nothink · GPT-5.4 mini·nothink · GPT-5.4·nothink · Gemini 3 Flash·minimal |
| arxiv-2604.26954-kappa-temp1-j7 arxiv-2604.26954 | — | xmodel | GPT-5.4 nano·nothink · GPT-5.4 mini·nothink · GPT-5.4·nothink · Gemini 3 Flash·minimal |
| arxiv-2609.29933-t1-longdocurl-agent-6-tool-fc arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t1-mmlongbench-doc-agent-6-tool-fc arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t1-mmlongbench-doc-all-images arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-longdocurl-ldurl-column-k-10 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-longdocurl-ldurl-column-k-2 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-longdocurl-ldurl-column-k-20 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-longdocurl-ldurl-column-k-5 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-10 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-2 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-20 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.29933-t12-mmlongbench-doc-extractor-mmlb-column-k-5 arxiv-2609.29933 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2609.33055-weighted-sd arxiv-2609.33055 | — | xmodel | Claude Sonnet 5·high · GPT-5.6 Luna·nothink · Gemini 3.8 Flash·medium |
| arxiv-2610.02204-capagent0-mean-success arxiv-2610.02204 | — | xmodel | Claude Opus 5·xhigh |
| arxiv-2610.04976-pricedguidance-dev-fable51-median-bits arxiv-2610.04976 | — | xmodel | Claude Fable 5.1·high · Claude Fable 5.1·max |
| arxiv-2610.04976-pricedguidance-dev-opus5-median-bits arxiv-2610.04976 | — | xmodel | Claude Opus 5·high · Claude Opus 5·max |
| arxiv-2610.04976-pricedguidance-test-median-bits arxiv-2610.04976 | — | xmodel | Claude Fable 5.1·max · Claude Opus 5·high · GPT-6 Astra·max · GLM-5.3·max |
| arxiv-2610.05348-hotpotqa-abst arxiv-2610.05348 | — | xmodel | Claude Sonnet 5.5·high |
| arxiv-2610.05348-nq-abst arxiv-2610.05348 | — | xmodel | Claude Sonnet 5.5·high |
| arxiv-2610.06668-tier2reject arxiv-2610.06668 | — | xmodel | Claude Fable 5.1·high · Claude Sonnet 5.5·high · Claude Opus 5·high · Claude Sonnet 5·high · Claude Opus 4.8·high · GPT-6 Sol·high · GPT-5.5·high |
| arxiv-2610.07563-householdbench-avg-rank arxiv-2610.07563 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high · Claude Opus 4.8·nothink · Qwen3.6-27B·nothink |
| arxiv-2610.12596-spirebench-acts arxiv-2610.12596 | — | xmodel | Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·max · Claude Sonnet 5·low · Claude Sonnet 5·max · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · GPT-6 Astra·low · GPT-6 Astra·max · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GLM-5.3·low · GLM-5.3·max · GLM-5.3-Flash·max · DeepSeek V4.1 Flash·max · Grok 4.6·xhigh |
| arxiv-2610.13075-longbench-v2-direct arxiv-2610.13075 | — | xmodel | Qwen3.8-27B·xhigh · Gemini 3.8 Flash·high · GPT-5.6 Luna·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GPT-5.6 Sol·max · GLM-5.3-Flash·max |
| arxiv-2610.13075-officeqa-v1-direct arxiv-2610.13075 | — | xmodel | Qwen3.8-27B·xhigh · Gemini 3.8 Flash·high · GPT-5.6 Luna·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GPT-5.6 Sol·max · GLM-5.3-Flash·max |
| arxiv-2610.13075-officeqa-v2-direct arxiv-2610.13075 | — | xmodel | Qwen3.8-27B·xhigh · Gemini 3.8 Flash·high · GPT-5.6 Luna·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GPT-5.6 Sol·max · GLM-5.3-Flash·max |
| arxiv-2610.14397-realitybench-doc arxiv-2610.14397 | — | xmodel | Gemini 3.8 Flash·medium · GPT-6 Sol·medium · GPT-6 Luna·medium · Claude Fable 5.1·high · Claude Opus 5·high · Grok 4.6·high · DeepSeek V4.1 Flash·high |
| arxiv-2610.17016-covert-kl-nats arxiv-2610.17016 | — | xmodel | Claude Sonnet 5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Terra·medium · Qwen3.8 Flash·nothink |
| arxiv-2610.17016-credit-kl-nats arxiv-2610.17016 | — | xmodel | Claude Sonnet 5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Terra·medium · Gemini 3.8 Flash·medium · Qwen3.8 Flash·nothink |
| arxiv-2610.17016-peer-kl-nats arxiv-2610.17016 | — | xmodel | Claude Sonnet 5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Terra·medium · Gemini 3.8 Flash·medium · Qwen3.8 Flash·nothink |
| arxiv-2610.18449-asb-rate arxiv-2610.18449 | — | xmodel | Claude Opus 5.5·max · GPT-5.6 Terra·max · GPT-5.6 Luna·max · Claude Sonnet 5.5·high |
| arxiv-2610.22599-direction arxiv-2610.22599 | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · GLM-5.3·max · Kimi K3·max |
| arxiv-2610.22599-r2 arxiv-2610.22599 | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · GLM-5.3·max · Kimi K3·max |
| arxiv-2610.22599-slope arxiv-2610.22599 | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · GLM-5.3·max · Kimi K3·max |
| arxiv-2610.22599-top-idea-gain arxiv-2610.22599 | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · GLM-5.3·max · Kimi K3·max |
| arxiv-2610.25078-homebench-im-f1 arxiv-2610.25078 | — | xmodel | GLM-5.3·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-homebench-im-succ arxiv-2610.25078 | — | xmodel | GLM-5.3·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-homebench-vm-f1 arxiv-2610.25078 | — | xmodel | GLM-5.3·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-homebench-vm-succ arxiv-2610.25078 | — | xmodel | GLM-5.3·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-bird-acf1-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-bird-safedeny-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-bird-safeex-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-bird-viol-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-spider-acf1-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-spider-safedeny-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-spider-safeex-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-rbac-spider-viol-direct arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-traceelephant-when arxiv-2610.25078 | — | xmodel | GLM-5.3·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-traceelephant-who arxiv-2610.25078 | — | xmodel | GLM-5.3·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-ww-algo-when arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-ww-algo-who arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-ww-hand-when arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-ww-hand-who arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-ww-pro-when arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25078-ww-pro-who arxiv-2610.25078 | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| arxiv-2610.25920-algoveri-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-algoveri-maestro arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl03-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl03-cc-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl03-maestro arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl03-maestro-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl06-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl06-cc-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl06-maestro arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-axl06-maestro-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fatex-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fatex-cc-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fatex-maestro arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fatex-maestro-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fatex-naive arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fqb-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fqb-cc-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-fqb-maestro-rep arxiv-2610.25920 | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-imo26-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high |
| arxiv-2610.25920-lib-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-lib-maestro arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-ntp4vc-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-ntp4vc-maestro arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-putnam-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-putnam-maestro arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.25920-verina-cc arxiv-2610.25920 | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·high · Claude Fable 5·high · GPT-5.6 Sol·high |
| arxiv-2610.26538-bymeal-breakfast arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-bymeal-dinner arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-bymeal-lunch arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-effort-high arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·high · Gemini 4 Argon·high |
| arxiv-2610.26538-effort-low arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-grams-eaten-relative-error arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t2-before-only arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t2-before-x-c arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t2-both-photos arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t2-both-valid arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t2-subtraction arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t3-1 arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t3-3 arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t3-3a arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t3-3b arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t5-within-800kj-before-only arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.26538-t5-within-800kj-both-photos arxiv-2610.26538 | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low |
| arxiv-2610.27043-t2-fb-de arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-fb-en arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-fb-es arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-fb-hi arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-ar arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-de arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-en arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-es arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-hi arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-it arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-ja arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27043-t2-mintaka-pt arxiv-2610.27043 | — | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high |
| arxiv-2610.27745-first-success arxiv-2610.27745 | — | xmodel | Gemini 3.8 Flash·low · Gemini 3.8 Flash·high |
| arxiv-2610.27745-h1 arxiv-2610.27745 | — | xmodel | Gemini 3.8 Flash·low · Gemini 3.8 Flash·high |
| arxiv-2610.27745-h2 arxiv-2610.27745 | — | xmodel | Gemini 3.8 Flash·low · Gemini 3.8 Flash·high |
| arxiv-2610.27745-h4a arxiv-2610.27745 | — | xmodel | Gemini 3.8 Flash·low · Gemini 3.8 Flash·high |
| arxiv-2610.27745-spread-prior arxiv-2610.27745 | — | xmodel | Gemini 3.8 Flash·low · Gemini 3.8 Flash·high |
| arxiv-2610.27745-spread-reported arxiv-2610.27745 | — | xmodel | Gemini 3.8 Flash·low · Gemini 3.8 Flash·high |
| arxiv-2610.27745-spread-rho arxiv-2610.27745 | — | xmodel | Gemini 3.8 Flash·low · Gemini 3.8 Flash·high |
| arxiv-2610.29207-t13-h3-d arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink |
| arxiv-2610.29207-t3-correct arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 5.5·medium |
| arxiv-2610.29207-t3-outdated arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 5.5·medium |
| arxiv-2610.29207-t3-uninterpretable arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 5.5·medium |
| arxiv-2610.29207-t3-wrong arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 5.5·medium |
| arxiv-2610.29207-t4-h1-d arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 5.5·medium |
| arxiv-2610.29207-t4-h2-s arxiv-2610.29207 | — | xmodel | Claude Sonnet 4.6·nothink · Claude Opus 5.5·medium |
| arxiv-2610.29207-t5-outdated-count arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 5.5·medium |
| arxiv-2610.29207-t8-correct arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink |
| arxiv-2610.29207-t8-outdated arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink |
| arxiv-2610.29207-t8-uninterpretable arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink |
| arxiv-2610.29207-t8-wrong arxiv-2610.29207 | — | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink |
| arxiv-2610.30051-libero10-success arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low · GPT-6 Astra·high · GPT-6 Astra·medium · GPT-6 Astra·low · GPT-6.1 Sol·high · GPT-6.1 Sol·medium · GPT-6.1 Sol·low · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high |
| arxiv-2610.30051-libero10-task0-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task1-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task2-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task3-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task4-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task5-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task6-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task7-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task8-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30051-libero10-task9-solved arxiv-2610.30051 | — | xmodel | Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high |
| arxiv-2610.30116-author-difficulty arxiv-2610.30116 | — | xmodel | Gemini 3.8 Flash·high · Qwen3.8-27B·xhigh |
| arxiv-2610.30116-author-hintlift arxiv-2610.30116 | — | xmodel | Gemini 3.8 Flash·high · Qwen3.8-27B·xhigh |
| arxiv-2610.30116-solve arxiv-2610.30116 | — | xmodel | Gemini 3.8 Flash·high · Qwen3.8-27B·xhigh |
| arxiv-2610.31986-t4-i1 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-i2 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-i3 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-ins arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-overall arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-p1 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-p2 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-p3 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-p4 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-perc arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-r1 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-r2 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-r3 arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.31986-t4-reas arxiv-2610.31986 | — | xmodel | Gemini 4 Argon·low · Gemini 3.8 Flash·low · Gemini 3.1 Pro·low · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash-Lite·minimal |
| arxiv-2610.34212-reference-verified-rate arxiv-2610.34212 | — | xmodel | GPT-5.5·low · GPT-5.5·high |
| arxiv-2610.34435-asr-all-allrev arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-all-gpt6sol arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-all-opus48 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-all-sonnet5 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-fix-allrev arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-fix-gpt6sol arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-fix-opus48 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-fix-sonnet5 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-window-allrev arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-window-gpt6sol arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-window-opus48 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-asr-window-sonnet5 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-all-glm-cc arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-all-glm-codex arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-all-gpt6sol arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-all-grok47 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-all-kimik3 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-all-opus48 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-all-sonnet5 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-all arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-glm-cc arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-glm-codex arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-gpt6sol arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-grok47 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-kimik3 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-opus48 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-fix-sonnet5 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-task-all arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-task-glm-cc arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-task-glm-codex arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-task-sonnet5 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-all arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-glm-cc arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-glm-codex arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-gpt6sol arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-grok47 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-kimik3 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-opus48 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-catch-window-sonnet5 arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.34435-reviewer-catch-rate arxiv-2610.34435 | — | xmodel | Claude Sonnet 5·high · Claude Opus 4.8·high |
| arxiv-2610.35671-cotcontrol-baseline arxiv-2610.35671 | — | xmodel | DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max |
| arxiv-2610.35671-cotcontrol-constrained arxiv-2610.35671 | — | xmodel | DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max |
| arxiv-2610.35671-ncri arxiv-2610.35671 | — | xmodel | DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·low · DeepSeek V4 Pro 0813 build·max |
| arxiv-2610.35671-paperbench-codedev arxiv-2610.35671 | — | xmodel | Gemini 3.8 Flash·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max |
| arxiv-2610.35671-rebench-rust arxiv-2610.35671 | — | xmodel | DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max |
| arxiv-2610.36434-geomean12-justgoal-single arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-geomean12-kda05-repro arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-dsa-heavy-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-dsa-heavy-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-dsa-mixed-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-dsa-mixed-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-dsa-small-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-dsa-small-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-fa4-bwd-gqa-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-fa4-bwd-gqa-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-fa4-bwd-mha-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-fa4-bwd-mha-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-fa4-bwd-varlen-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-fa4-bwd-varlen-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-gdn-few-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-gdn-few-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-gdn-long-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-gdn-long-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-gdn-short-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-gdn-short-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-moe-medium-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-moe-medium-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-moe-prefill-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-moe-prefill-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-moe-token-justgoal arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t7-moe-token-kda05 arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t8-dsa-fwd-justgoal-whole arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t8-fa4-bwd-justgoal-whole arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t8-gdn-fwd-justgoal-whole arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.36434-t8-moe-fp8-justgoal-whole arxiv-2610.36434 | — | xmodel | Claude Opus 4.8·high · Claude Opus 5·high · Claude Fable 5.1·high |
| arxiv-2610.37747-found-likert arxiv-2610.37747 | — | xmodel | Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high |
| arxiv-2610.38096-s14-robust-beams-rejected-bt arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s14-robust-beams-rejected-bu-retest arxiv-2610.38096 | — | xmodel | Gemini 3.8 Flash·high · Claude Opus 5·high |
| arxiv-2610.38096-s14-robust-beams-rejected-bx arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s14-robust-beams-rejected-c4-bu arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-c4-u-accepted arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-c4-u-flagged arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-c4-u-noans arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-c4-u-rejected arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-c4-u-solved arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-c4-u-vnoans arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-c4-u-wrong arxiv-2610.38096 | — | xmodel | Claude Haiku 5.5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s2-t02-fine-u-accepted arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s2-t02-fine-u-rejected arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s2-t02-fine-u-solved arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s2-t02-main-t-accepted arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-t-flagged arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-t-rejected arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-t-solved arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-t-wrong arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-u-accepted arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-u-flagged arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-u-noans arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-u-vnoans arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-u-wrong arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-main-x-accepted arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t02-main-x-flagged arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t02-main-x-noans arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t02-main-x-rejected arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t02-main-x-solved arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t02-main-x-vnoans arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t02-main-x-wrong arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t02-pilot-u-accepted arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-pilot-u-rejected arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t02-pilot-u-solved arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-fine-u-rejected arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s2-t11-main-t-accepted arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-t-flagged arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-t-rejected arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-t-solved arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-u-accepted arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-u-flagged arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-u-noans arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-u-vnoans arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-u-wrong arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-main-x-accepted arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t11-main-x-flagged arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t11-main-x-noans arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t11-main-x-rejected arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t11-main-x-solved arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t11-main-x-vnoans arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t11-main-x-wrong arxiv-2610.38096 | — | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| arxiv-2610.38096-s2-t11-pilot-u-rejected arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s2-t11-pilot-u-solved arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s33-t02-feasibility-only-valid-solved arxiv-2610.38096 | — | xmodel | Claude Opus 5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s33-t02-pa-limit-valid-solved arxiv-2610.38096 | — | xmodel | Claude Opus 5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s33-t02-pa-note-rejected arxiv-2610.38096 | — | xmodel | Claude Opus 5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s33-t02-pa-note-valid-solved arxiv-2610.38096 | — | xmodel | Claude Opus 5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s33-t02-s1-limit-valid-solved arxiv-2610.38096 | — | xmodel | Claude Opus 5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s33-t02-s1-note-rejected arxiv-2610.38096 | — | xmodel | Claude Opus 5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s33-t02-s1-note-valid-solved arxiv-2610.38096 | — | xmodel | Claude Opus 5·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-s33-t11-feasibility-only-valid-solved arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5·high |
| arxiv-2610.38096-s33-t11-pa-limit-valid-solved arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5·high |
| arxiv-2610.38096-s33-t11-pa-note-rejected arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5·high |
| arxiv-2610.38096-s33-t11-pa-note-valid-solved arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5·high |
| arxiv-2610.38096-s33-t11-s1-limit-valid-solved arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5·high |
| arxiv-2610.38096-s33-t11-s1-note-rejected arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5·high |
| arxiv-2610.38096-s33-t11-s1-note-valid-solved arxiv-2610.38096 | — | xmodel | GPT-6 Luna·high · Claude Opus 5·high |
| arxiv-2610.38096-s7-t02-fine-max-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s7-t02-fine-median-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s7-t02-main-capable arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Luna·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t02-main-max-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t02-main-median-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t02-pilot-max-error arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t02-pilot-median-error arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t11-fine-max-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s7-t11-fine-median-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| arxiv-2610.38096-s7-t11-main-max-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t11-main-median-error arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t11-pilot-max-error arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-s7-t11-pilot-median-error arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Claude Opus 4.8·high |
| arxiv-2610.38096-t1-beam arxiv-2610.38096 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Claude Opus 5·high |
| arxiv-2610.38096-t1-disk arxiv-2610.38096 | — | xmodel | Gemini 3.8 Flash·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Claude Haiku 5.5·high · Claude Opus 5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-t1-noans arxiv-2610.38096 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Haiku 5.5·high |
| arxiv-2610.38096-t1-probes arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Luna·high · Claude Opus 4.8·high |
| arxiv-2610.38096-t1-robust arxiv-2610.38096 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high · Claude Haiku 5.5·high · Claude Opus 5·high |
| arxiv-2610.38096-t1-vbeam arxiv-2610.38096 | — | xmodel | Gemini 4 Argon·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-t1-vdisk arxiv-2610.38096 | — | xmodel | GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · Claude Opus 4.8·high |
| arxiv-2610.38096-t3-t02-large arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high |
| arxiv-2610.38096-t3-t02-near arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high |
| arxiv-2610.38096-t3-t02large arxiv-2610.38096 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.38096-t3-t11-large arxiv-2610.38096 | — | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high |
| arxiv-2610.38249-data arxiv-2610.38249 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2610.38249-knowledge arxiv-2610.38249 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2610.38249-mean arxiv-2610.38249 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2610.38249-workout arxiv-2610.38249 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2610.38924-a-t2-pass1 arxiv-2610.38924 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.38924-a-t2-pass3 arxiv-2610.38924 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.38924-a-t2-passhat3 arxiv-2610.38924 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.38924-a-t3-firstread arxiv-2610.38924 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.38924-a-t3-images arxiv-2610.38924 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.38924-a-t3-ocr arxiv-2610.38924 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.38924-a-t3-upload arxiv-2610.38924 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.39438-t3-approp arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh |
| arxiv-2610.39438-t3-cr arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh · GLM-5.3·max |
| arxiv-2610.39438-t3-csr arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh · GLM-5.3·max |
| arxiv-2610.39438-t3-evidence arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh · GLM-5.3·max |
| arxiv-2610.39438-t3-factual arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh |
| arxiv-2610.39438-t3-intent arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh · GLM-5.3·max |
| arxiv-2610.39438-t3-strategy arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh · GLM-5.3·max |
| arxiv-2610.39438-t3-success-k arxiv-2610.39438 | — | xmodel | GPT-5.6 Luna·medium · GPT-6 Luna·xhigh |
| arxiv-2610.40503-arcagi3-public-standard arxiv-2610.40503 | — | xmodel | Claude Opus 5·high · GPT-6 Astra·max |
| arxiv-2610.41025-argon-autopsy-fixrate arxiv-2610.41025 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max · Grok 4.7·xhigh |
| arxiv-2610.41025-argon-autopsy-fullyfixed arxiv-2610.41025 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2610.41025-argon-autopsy-repair arxiv-2610.41025 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.41025-argon-autopsy-valid arxiv-2610.41025 | — | xmodel | Gemini 4 Argon·high · Gemini 3.8 Flash·high |
| arxiv-2610.41025-sup-effect-failing arxiv-2610.41025 | — | xmodel | Gemini 3.8 Flash·high · GPT-6 Luna·max · GPT-6.1 Sol·max |
| arxiv-2610.41025-sup-effect-succeeding arxiv-2610.41025 | — | xmodel | Gemini 3.8 Flash·high · GPT-6 Luna·max · GPT-6.1 Sol·max |
| arxiv-2610.41025-t4-net-all arxiv-2610.41025 | — | xmodel | Gemini 3.8 Flash·high · GPT-6 Luna·max · GPT-6.1 Sol·max |
| arxiv-2610.41025-t4-net-nonqwen arxiv-2610.41025 | — | xmodel | Gemini 3.8 Flash·high · GPT-6 Luna·max · GPT-6.1 Sol·max |
| arxiv-2610.41076-baba-difficulty-hard arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Grok 4.6·high |
| arxiv-2610.41076-baba-difficulty-light arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Grok 4.6·high |
| arxiv-2610.41076-baba-difficulty-standard arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Grok 4.6·high |
| arxiv-2610.41076-baba-issue-completion arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Grok 4.6·high |
| arxiv-2610.41076-baba-pass1-strict arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Grok 4.6·high |
| arxiv-2610.41076-baba-pass3 arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high · Grok 4.6·high |
| arxiv-2610.41076-t13-anyseed arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t14-hard arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t14-light arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t14-std arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t15-seed1-outcome arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t15-seed1-strict arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t15-seed2-outcome arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t15-seed2-strict arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t15-seed3-outcome arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2610.41076-t15-seed3-strict arxiv-2610.41076 | — | xmodel | Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.1 Pro·high |
| arxiv-2611.02075-fewshot arxiv-2611.02075 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2611.02075-zeroshot arxiv-2611.02075 | — | xmodel | Qwen3.5-27B·nothink |
| arxiv-2611.03698-t1-lite-avg arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-lite-qwen-1-7b arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-lite-qwen-4b arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-lite-smollm3-3b arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-ptb-avg arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-ptb-gemma3-4b arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-ptb-qwen-1-7b arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-ptb-qwen-4b arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t1-ptb-smollm3-3b arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t2-at-least-one-successful-run arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t2-eff-total-rate arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.03698-t2-improved arxiv-2611.03698 | — | xmodel | GPT-5.4·medium · GPT-5.5·medium |
| arxiv-2611.05147-gm-league-arena-baseball-cash arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-baseball-elo arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high · Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-football-cash arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Luna·high · GPT-5.6 Terra·high · GPT-5.4 mini·high · GPT-5.4 nano·high |
| arxiv-2611.05147-gm-league-arena-football-elo arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-6 Luna·high · GPT-5.6 Terra·high · GPT-5.4 mini·high · GPT-5.4 nano·high · Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-hockey-cash arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-5.4 mini·high · GPT-5.6 Terra·high · GPT-5.4 nano·high · GPT-6 Luna·high |
| arxiv-2611.05147-gm-league-arena-hockey-elo arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6.1 Sol·high · GPT-6 Astra·high · GPT-5.4 mini·high · GPT-5.6 Terra·high · GPT-5.4 nano·high · GPT-6 Luna·high · Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-nba-cash arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-nba-elo arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high · Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-nba-gi arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high · Grok 4.7·high · Grok 4.6·high |
| arxiv-2611.05147-gm-league-arena-nba-losses arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-nba-wins arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-nba-wins-2022 arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-nba-wins-2023 arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-nba-wins-2024 arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-nba-wins-2025 arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.05147-gm-league-arena-nba-wins-2026 arxiv-2611.05147 | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·high · GPT-6 Astra·high · GPT-6.1 Sol·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| arxiv-2611.06135-p1-f1-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-f1-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-fn-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-fn-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-fp-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-fp-pat arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-fp-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-hit-sample49 arxiv-2611.06135 | — | xmodel | DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·nothink |
| arxiv-2611.06135-p1-hit-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·high |
| arxiv-2611.06135-p1-mcc-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·high |
| arxiv-2611.06135-p1-mcc-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·high |
| arxiv-2611.06135-p1-tp-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p1-tp-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| arxiv-2611.06135-p2-conf1-pat arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-conf1-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-f1-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-f1-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-fn-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-fn-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-fp-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-fp-pat arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-fp-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-hit-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·high |
| arxiv-2611.06135-p2-mcc-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-mcc-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·high |
| arxiv-2611.06135-p2-tp-all arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-p2-tp-vul arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-c-f1 arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-c-mcc arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-c-pc arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-f1 arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-mcc arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-pb arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-pc arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-pr arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-pv arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-rr-f1 arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-rr-mcc arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06135-t5-rr-pc arxiv-2611.06135 | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh |
| arxiv-2611.06841-serverbench-dpd-gemma-e4b arxiv-2611.06841 | — | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5·max |
| arxiv-2611.06841-serverbench-dpd-qwen35-9b arxiv-2611.06841 | — | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5·max |
| arxiv-2611.06841-serverbench-dpd-qwen38-27b-int4 arxiv-2611.06841 | — | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5·max |
| arxiv-2611.07461-37-flash-t13-baseline-resolution arxiv-2611.07461 | — | xmodel | Gemini 3.8 Flash·high · Gemini 3.7 Flash·high |
| arxiv-2611.07461-37-flash-t13-delta arxiv-2611.07461 | — | xmodel | Gemini 3.8 Flash·high · Gemini 3.7 Flash·high |
| arxiv-2611.07461-t12-delta-mini arxiv-2611.07461 | — | xmodel | Claude Fable 5·medium · Claude Opus 5·medium · Claude Sonnet 5·medium · Claude Opus 4.8·medium · Claude Opus 4.7·medium |
| arxiv-2611.07461-t12-hermes-resolved arxiv-2611.07461 | — | xmodel | Claude Fable 5·medium · Claude Opus 5·medium · Claude Sonnet 5·medium · Claude Opus 4.8·medium · Claude Opus 4.7·medium |
| arxiv-2611.07461-t12-mini-resolved arxiv-2611.07461 | — | xmodel | Claude Fable 5·medium · Claude Opus 5·medium · Claude Sonnet 5·medium · Claude Opus 4.8·medium · Claude Opus 4.7·medium |
| arxiv-2611.07461-t13-baseline-resolution arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Opus 5·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·medium · Claude Sonnet 5·max |
| arxiv-2611.07461-t13-delta arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Opus 5·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·medium · Claude Sonnet 5·max |
| arxiv-2611.07461-t13-grokbuild-baseline-resolution arxiv-2611.07461 | — | xmodel | Grok 4.6·high · Grok 4.5·high |
| arxiv-2611.07461-t13-hermes-resolution arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Opus 5·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·medium · Claude Sonnet 5·max · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Grok 4.6·high · Grok 4.5·high |
| arxiv-2611.07461-t14-baseline-avg arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-baseline-build arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-baseline-issue arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-baseline-monitor arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-baseline-test arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-delta arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-hermes-avg arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-hermes-build arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-hermes-issue arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-hermes-monitor arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t14-hermes-test arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Sonnet 5·medium |
| arxiv-2611.07461-t31-delta arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Opus 5·high · Claude Sonnet 5·medium · Claude Sonnet 5·high |
| arxiv-2611.07461-t31-hermes-comp arxiv-2611.07461 | — | xmodel | Claude Opus 5·medium · Claude Opus 5·high · Claude Sonnet 5·medium · Claude Sonnet 5·high |
| arxiv-2611.07667-auction-coarse arxiv-2611.07667 | — | xmodel | Grok 4.7·low · GPT-5.1·low |
| arxiv-2611.07667-auction-full arxiv-2611.07667 | — | xmodel | Grok 4.7·low · GPT-5.1·low |
| arxiv-2611.07667-bargaining-coarse arxiv-2611.07667 | — | xmodel | GPT-6 Sol·low · GPT-5.1·low · Grok 4.7·low |
| arxiv-2611.07667-bargaining-full arxiv-2611.07667 | — | xmodel | GPT-6 Sol·low · GPT-5.1·low · Grok 4.7·low |
| arxiv-2611.07667-doubleauction-coarse arxiv-2611.07667 | — | xmodel | Claude Opus 5.5·low · Gemini 3.1 Pro·low |
| arxiv-2611.07667-doubleauction-full arxiv-2611.07667 | — | xmodel | Claude Opus 5.5·low · Gemini 3.1 Pro·low |
| arxiv-2611.07667-trade-exact arxiv-2611.07667 | — | xmodel | GPT-6 Sol·low · GPT-6 Luna·low · GPT-5.1·low |
| arxiv-2611.07667-trade-ordinal arxiv-2611.07667 | — | xmodel | GPT-6 Sol·low · GPT-6 Luna·low · GPT-5.1·low |
| arxiv-2611.07962-tcsexam-full-constructions arxiv-2611.07962 | — | xmodel | GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-index arxiv-2611.07962 | — | xmodel | GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-open-conjectures arxiv-2611.07962 | — | xmodel | GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-solver arxiv-2611.07962 | — | xmodel | GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-full-theorems arxiv-2611.07962 | — | xmodel | GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-constructions arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-index arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-open-conjectures arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-solver arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-lite-theorems arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Grok 4.6·xhigh |
| arxiv-2611.07962-tcsexam-t25-corners-theorem arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t25-discrete-bidding-games arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t25-folded-rs-codes arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t25-quantum-cryptography arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t25-small-depth-negations arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t26-bin-packing arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t26-course-timetabling arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t26-machine-reassignment arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t26-maximum-cut arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t26-quantified-boolean arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t28-branching-programs arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t28-evasiveness arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t28-frege-proof-length arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t28-log-rank arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.07962-tcsexam-t28-online-matching arxiv-2611.07962 | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max |
| arxiv-2611.08209-roboarcade-success-multiview-rgbd arxiv-2611.08209 | — | xmodel | Claude Fable 5.1·xhigh · GPT-6 Astra·xhigh |
| arxiv-2611.08209-roboarcade-success-via arxiv-2611.08209 | — | xmodel | Claude Opus 5·xhigh · Claude Fable 5.1·xhigh · GPT-5.6 Sol·xhigh · GPT-6 Astra·xhigh |
| arxiv-2611.08209-roboarcade-t6-so-cost-via arxiv-2611.08209 | — | xmodel | Claude Opus 5·xhigh · Claude Fable 5.1·xhigh |
| arxiv-2611.08472-xstack-pass-cheat arxiv-2611.08472 | — | xmodel | Claude Opus 5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high |
| arxiv-2611.08472-xstack-pass-correctness arxiv-2611.08472 | — | xmodel | Claude Opus 5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high |
| arxiv-2611.08472-xstack-pass-target arxiv-2611.08472 | — | xmodel | Claude Opus 5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high |
| arxiv-2611.08472-xstack-pass-weights arxiv-2611.08472 | — | xmodel | Claude Opus 5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high |
| arxiv-2611.09017-frontier3dbench-b2s arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-i2c arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-i2c-sv arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-i2e arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-l1 arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-l2 arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-l3 arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-r2s arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-r2s-rp arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-se arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-t2c arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-t2e arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09017-frontier3dbench-total arxiv-2611.09017 | — | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max |
| arxiv-2611.09802-t2-location-corrected-fixed arxiv-2611.09802 | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium |
| arxiv-2611.09802-t2-location-removed-fixed arxiv-2611.09802 | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium |
| arxiv-2611.09802-t2-location-stale-fixed arxiv-2611.09802 | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium |
| arxiv-2611.09802-t2-procedure-corrected-fixed arxiv-2611.09802 | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium |
| arxiv-2611.13246-expertcrowd-swe-macro-score arxiv-2611.13246 | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max |
| arxiv-2611.16591-inference-income arxiv-2611.16591 | — | xmodel | Claude Opus 5.5·medium · Claude Opus 5.5·high · GPT-6.1 Sol·max |
| arxiv-2611.16591-inference-income-multiagent arxiv-2611.16591 | — | xmodel | Claude Opus 5.5·high · GPT-6.1 Sol·max |
| arxiv-2611.16591-singularity-index arxiv-2611.16591 | — | xmodel | Claude Opus 5.5·max · GPT-6.1 Sol·max |
| arxiv-2611.17955-webarenapro-sr arxiv-2611.17955 | — | xmodel | Claude Opus 5.5·high · GPT-6.1 Sol·high |
| arxiv-2611.18562-libero-success arxiv-2611.18562 | — | xmodel | GPT-6.1 Sol·medium · GPT-6.1 Sol·max · GPT-6 Astra·medium · GPT-6 Astra·max |
| Atomic Agent expense-lib atomicagent | Atomic Agent 0.6.5 via OpenRouter · 5 bugs + 2 features, 32 hidden tests, n=3, provider default effort, billed cost; saturated ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5·high · GPT-6 Sol·medium |
| atomicagent-warehouse-59-hidden-tests atomicagent | — | xmodel | Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Opus 5.5·high |
| AttractorBench v2.0.0 NLSpec (T1/T2/T3+judge), codex harness, best run per model strongdm-attractorbench | strongdm/attractorbench bench_version 2.0.0, main task only; score=5% build+5% self-test+25% T1+25% T2+25% T3+15% judge; Harbor, agent codex; leaderboard statistic = best run per agent/model (snapshot 2026-02-26); cost = Harbor result.json x litellm pricing; read 2026-10-20 ✓ | xmodel | GPT-5.3-Codex·high · GPT-5.2·high |
| AutoBCI (2609.35456) — agentic EEG architecture search, Claude Code, 6-round search arxiv-2609.35456 | Table 3, average test bAcc%/wF1/Kappa over MI/emotion/sleep held-out sets; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Gemini 3.8 Flash·medium |
| AutomationB. arxiv-2604.18934 | harness+effort NOT stated ✓ | xmodel | Claude Opus 4.7·max · Gemini 3.1 Pro·high · GPT-5.4·high · GPT-5.4·nothink |
| AutomationBench (OpenAI GPT-5.6 post chart) openai | AutomationBench pass rate, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · Gemini 3.5 Flash·high · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Claude Opus 4.8·max |
| AutomationBench (OpenAI GPT-6 Astra post chart, non-code mode) openai | AutomationBench; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max |
| AutomationBench (OpenAI GPT-6.1 Sol post) openai | AutomationBench; OpenAI GPT-6.1 Sol post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; Wayback first capture; read 2026-10-19 ✓ | xmodel | GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max |
| Bakery website, Sonnet 5.5 vs Opus 5.5 at high (AI Field Notes video) youtube-aifieldnotes | Claude Code; same brief; empty folder; high effort; one try; 2026-10-10; read 2026-10-18 ✓ | xmodel | Claude Sonnet 5.5·high · Claude Opus 5.5·high · Claude Haiku 5.5·high |
| bangla-llm-bench, belebele track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Gemini 3.8 Flash·medium · DeepSeek V4.1 Flash·high |
| bangla-llm-bench, bnexam track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Gemini 3.8 Flash·medium · DeepSeek V4.1 Flash·high |
| bangla-llm-bench, mgsm track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=250 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Gemini 3.8 Flash·medium · DeepSeek V4.1 Flash·high |
| bangla-llm-bench, mmlu track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=285 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Gemini 3.8 Flash·medium · DeepSeek V4.1 Flash·high |
| Bazaar multi-attribute auction, cumulative profit bazaar-2608.00102 | Bazaar sealed-bid multi-attribute auction, ~79 rounds, 10 seeds per row (GPT-5.5 xhigh 3 seeds), closed-form evaluation, no tools beyond bidding API; metric cumulative profit USD (game money, not spend; efficiency eta of Table 6 = same runs, not kept); arXiv 2608.00102 Table 3 (Aug 2026); read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·high · GPT-5.5·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·medium · GPT-5.4·medium |
| BenchCAD (python tool) (OpenAI GPT-6 Astra post chart, non-code mode) openai | BenchCAD (python tool); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| benchlm-aa-agentic-index benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-aa-analystagent benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-coding-index benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-aa-gpqa-diamond benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high |
| benchlm-aa-hle benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-lcr benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-mmmu-pro benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high |
| benchlm-aa-omniscience-accuracy benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-omniscience-hallucination-rate benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-omniscience-index benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-scicode benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-aa-tau3-banking benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-arc-agi-1 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-arc-agi-2 benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-arc-agi-3 benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-artificial-analysis-intelligence-index benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-bug-hunt-bench benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-critpt benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-cursorbench-3-2 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high |
| benchlm-cursorbench-4-0 benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-cwe-bench-v1 benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-deepswe benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-design-arena-website benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-frontiermath-v2-tier-4 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-frontiermath-v2-tiers-1-3 benchlm | — | xmodel | Grok 4.5·high · Grok 4.6·high · Grok 4.7·high |
| benchlm-frontierswe-v2 benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-gpqa-diamond-vals benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-livecodebench-vals benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-mazebench benchlm | — | xmodel | Grok 4.6·high · Grok 4.7·high |
| benchlm-mmlu-pro-vals benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-overall-score benchlm | — | xmodel | Grok 4.1 Fast·nothink · Grok 4.20·think · Grok 4.5·high · Grok 4.6·high |
| benchlm-swe-bench-pro benchlm | — | xmodel | Grok 4.20·think · Grok 4.5·high |
| benchmarklist-aa-lcr-cost-vs-score-213-models benchmarklist | — | xmodel | Grok 4.3·nothink · Grok 4.3·low · Grok 4.3·medium · Grok 4.3·high |
| benchmarklist-arc-agi-1-of-105 benchmarklist | — | xmodel | DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max |
| benchmarklist-arc-agi-2-of-109 benchmarklist | — | xmodel | DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max |
| benchmarklist-arc-agi-2-of-115 benchmarklist | — | xmodel | Grok 4.6·xhigh · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·high · Grok 4.7·xhigh · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high |
| benchmarklist-artificial-analysis-intelligence-index-of-325 benchmarklist | — | xmodel | Grok 4.3·high · Grok 4.3·nothink |
| benchmarklist-deepswe-1-1-of-63 benchmarklist | — | xmodel | Grok 4.6·low · Grok 4.6·high · Grok 4.6·xhigh · Grok 4.6·medium |
| benchmarklist-frontiercode-2-0-of-26 benchmarklist | — | xmodel | Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high |
| benchmarklist-gpqa-diamond-of-481 benchmarklist | — | xmodel | Grok 4.3·high · Grok 4.3·nothink · Grok 4.3·low · Grok 4.3·medium |
| benchmarklist-itsmbench-of-5 benchmarklist | — | xmodel | Grok 4.5·low · Grok 4.5·medium |
| benchmarklist-obviousbench-of-275 benchmarklist | — | xmodel | Grok 4.3·high · Grok 4.3·nothink · Grok 4.3·low · Grok 4.3·medium · Grok 4.5·high · Grok 4.5·low · Grok 4.5·medium · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh |
| Binder design (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Sonnet 5·max · Claude Fable 5.1·max |
| Bio-imaging (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Sonnet 5·max · Claude Fable 5.1·max · Gemini 3.1 Pro·high · GPT-6 Astra·max |
| BioMystery hard-dist (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · Gemini 3.1 Pro·high · GPT-6 Astra·max |
| BioMystery hard-sub (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · Gemini 3.1 Pro·high · GPT-6 Astra·max |
| Bito agentic coding battery bito.ai | Bito's internal agentic coding battery in Claude Code (6 task types, multi-turn sessions), each turn scored out of 50 by Opus 5.5 · mean score /50, cost per session (USD); effort from the post; read 2026-10-01 ✓ | xmodel | Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high |
| bito-s55o55-pct bito.ai | — | xmodel | Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| BixBench (table) openai | BixBench; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| BixBench3 mean artifact score (20 comp-bio tasks) bixbench3-2608.25286 | BixBench3 20 tasks / 138 artifacts, Inspect AI ReAct agent, 5 tools, 24 h / 5000 message cap, programmatic artifact grading, 20 runs per model (260 runs), each model at its maximum thinking effort; metric mean task score 0-1; cost = mean USD per task over 20 tasks from per-call tokens x provider list prices; arXiv 2608.25286 (Aug 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh · Gemini 3.5 Flash·high · Claude Opus 5·max · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh · Gemini 3.1 Pro·high · Gemini 3.1 Flash-Lite·high |
| blog-minimax-m31f-automation minimax | — | xmodel | DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high |
| blog-minimax-m31f-deepswe minimax | — | xmodel | DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high |
| blog-qwen3.8: $OneMillion-Bench (expert score) alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark '$OneMillion-Bench (expert score)'; no cost or tokens in the source (score only); footnotes: "Evaluated using gemini-3.1-pro-preview."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: Agents' Last Exam (Pass / Score) [pass] alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Agents' Last Exam (Pass / Score)' [pass]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: AndroidBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'AndroidBench'; no cost or tokens in the source (score only); footnotes: "Evaluated on the 95-task public subset, reporting avg@3 scores."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: Automation-Bench (Pass@1) alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Automation-Bench (Pass@1)'; no cost or tokens in the source (score only); footnotes: "Evaluated on the 600-task public subset."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: CoWorkBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'CoWorkBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: DeepSWE 1.1 alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'DeepSWE 1.1'; no cost or tokens in the source (score only); footnotes: "We report the highest score among both harnesses"; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: GPQA Diamond alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'GPQA Diamond'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: HealthBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HealthBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: HLE alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HLE'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: HLE w/ tools alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HLE w/ tools'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: IFBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'IFBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: JobBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'JobBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: LongBench v2 alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'LongBench v2'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: MLS-Bench-Lite alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'MLS-Bench-Lite'; no cost or tokens in the source (score only); footnotes: "All other model scores are taken from the official leaderboard." / "Evaluated with Claude Code using a 5-hour timeout and max_tokens=131,072."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: MRCR v2 256K (8-needle) alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'MRCR v2 256K (8-needle)'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: PaperBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PaperBench'; no cost or tokens in the source (score only); footnotes: "Evaluated in the BasicAgent setting under Code-Dev mode, judged by Claude Opus 4.6, and averaged over 3 runs (max 12 hours per run)."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: PLawBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PLawBench'; no cost or tokens in the source (score only); footnotes: "Evaluated using gemini-3.1-pro-preview."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: PRBench-Finance alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PRBench-Finance'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: PRBench-Legal alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PRBench-Legal'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: QwenQoderBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenQoderBench'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness. Reporting avg@5 with a 6-hour timeout"; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: QwenReactBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenReactBench'; no cost or tokens in the source (score only); footnotes: "Inhouse React project building benchmark using Claude Code as the harness"; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: QwenSVGBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenSVGBench'; no cost or tokens in the source (score only); footnotes: "Inhouse SVG code generation benchmark; bilingual (EN/CN), auto-render + multimodal judge; BT/Elo rating."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: QwenSWEBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenSWEBench'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness. Reporting avg@3 with an 8-hour timeout"; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: SkillsBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'SkillsBench'; no cost or tokens in the source (score only); footnotes: "Opus 4.8 and Fable 5 are evaluated on Claude Code" / "GPT-5.6 Sol is evaluated on Codex" / "the Qwen-series are evaluated on OpenCode. All results are from our own testing."; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: Toolathlon Verified (Pass@1) alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Toolathlon Verified (Pass@1)'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| blog-qwen3.8: WorkSpaceBench alibaba | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'WorkSpaceBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Qwen3.8 Max·xhigh |
| Brood War Bench broodwarbench | Claude Code CLI · round-robin StarCraft:Brood War tournament, 171 games, default effort only ✓ | xmodel | Grok 4.6·xhigh · Grok 4.6·medium · Grok 4.6·low |
| BrowseComp (OpenAI GPT-5.6 post chart) openai | BrowseComp pass@1, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| BrowseComp (table) openai | BrowseComp; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| bsb-v1-green github-bullshitbench-v1 | — | xmodel | Claude Opus 4.8·nothink · Claude Sonnet 4.6·high · Claude Sonnet 5·low · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Opus 5·xhigh · Claude Sonnet 4.6·nothink · Claude Opus 4.8·xhigh · Claude Haiku 4.5·nothink · Claude Opus 4.7·nothink · Claude Fable 5.1·low · Claude Opus 5·low · Grok 4.6·low · Claude Fable 5.1·max · Grok 4.6·xhigh · Grok 4.5·high · Claude Opus 5.5·low · Gemini 3.5 Flash-Lite·minimal · GPT-6 Astra·max · GLM-5.3·low · Grok 4.5·low · DeepSeek V4.1 Flash·max · GLM-5.3·max · Gemini 3.5 Flash-Lite·high · Claude Opus 5.5·max · DeepSeek V4.1 Flash·low · GPT-6 Sol·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6 Luna·nothink · GPT-6 Astra·low · GPT-6 Sol·nothink · GPT-5.4·nothink · GPT-5.5·nothink · GPT-5.6 Sol·max · GPT-5.6 Sol·low · GPT-5.6 Terra·low · GPT-5.2·nothink · GPT-5.4 mini·xhigh · DeepSeek V4 Pro 0813 build·max · GPT-5.4 mini·high · GPT-5.4·xhigh · GPT-5.6 Luna·low · GPT-5.6 Luna·max · DeepSeek V4 Pro 0813 build·low · GPT-5.4 mini·nothink · Qwen3.8 Max·xhigh · GPT-5.2·high · Qwen3.8-27B·xhigh · GPT-5.4 nano·nothink · GPT-5.4 nano·high · Gemini 3.1 Pro·low · Gemini 3.7 Flash·low · GPT-5.4 nano·xhigh · Gemini 3 Pro·high · Gemini 3.7 Flash·high · GLM-5·think · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Grok 4.1 Fast·think · Gemini 3.6 Flash·minimal · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · Gemini 3 Flash·high · Grok 4.1 Fast·nothink · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high |
| bullshitbench-v1-strict github-cc-83510 | — | xmodel | Claude Sonnet 4.6·high · Claude Sonnet 5·low · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Opus 5·xhigh |
| BVB arxiv-2609.15478 | arXiv 2609.15478 · Mini-BVB via LiteLLM, $3/scene cap, 5 Claude models at high (+ omitted-effort runs), LiteLLM list cost ✓ | xmodel | GPT-6 Astra·xhigh · GPT-6.1 Sol·xhigh · GPT-6 Sol·xhigh · GPT-6 Astra·high · Claude Sonnet 5.5·xhigh · Claude Opus 5.5·xhigh · GPT-6 Luna·xhigh · GPT-5.6 Sol·xhigh · Grok 4.6·xhigh · Claude Opus 5·high · Claude Opus 5.5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·high · GPT-5.5·high · GPT-5.6 Sol·nothink · GPT-5.6 Sol·medium · GPT-5.6 Luna·high · GPT-5.6 Sol·low · Grok 4.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.6 Terra·medium · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · GPT-5.6 Luna·medium · GPT-5.2·high · GPT-5.6 Terra·low · Claude Sonnet 5·high · Claude Sonnet 4.6·high · GPT-5.6 Luna·low · Claude Opus 4.7·high · Gemini 3 Flash·high · Claude Opus 4.8·high · GPT-5.2·nothink · GPT-5.5·nothink · GPT-5.4·nothink · GPT-5.4 mini·nothink · Grok 4.3·high · Grok 4.3·nothink · Qwen3.5-27B·nothink |
| c0ze/agent-evaluation: 13 tasks, blind-graded (24-way pass of 13 Oct) c0ze-agent-evaluation | 13 self-contained tasks, 1 run per model, blind rubric grading by Opus 5.5 subagents; score = mean of 13 task totals /10; cost per 13-task run (credit-balance drop, whole $; Haiku API-equivalent); read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·xhigh · Claude Fable 5.1·xhigh · Claude Opus 5.5·high · Claude Haiku 5.5·xhigh · Grok 4.7·xhigh · GPT-6 Astra·xhigh · GPT-6.1 Sol·xhigh · GPT-6 Luna·max · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high |
| CAISI evaluation of DeepSeek V4 Pro: FrontierScience nist-caisi-deepseek-v4-pro | CAISI/NIST 'CAISI Evaluation of DeepSeek V4 Pro' (page of 1 May 2026, updated 2 May; evaluation April 2026): FrontierScience, accuracy % of tasks solved; agentic runs with Inspect ReAct agent (1M weighted-token budget for PortBench/CTF, 500k for SWE-Bench Verified); CAISI-served DeepSeek V4 on H200/B200 GPUs with developer-recommended settings incl. maximum thinking; cost (7 benchmarks) = Figure 4 average expense per solved item over the items BOTH GPT-5.4 mini and DeepSeek V4 Pro solved; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4 mini·xhigh |
| CAISI evaluation of DeepSeek V4 Pro: GPQA-Diamond nist-caisi-deepseek-v4-pro | CAISI/NIST 'CAISI Evaluation of DeepSeek V4 Pro' (page of 1 May 2026, updated 2 May; evaluation April 2026): GPQA-Diamond, accuracy % of tasks solved; agentic runs with Inspect ReAct agent (1M weighted-token budget for PortBench/CTF, 500k for SWE-Bench Verified); CAISI-served DeepSeek V4 on H200/B200 GPUs with developer-recommended settings incl. maximum thinking; cost (7 benchmarks) = Figure 4 average expense per solved item over the items BOTH GPT-5.4 mini and DeepSeek V4 Pro solved; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4 mini·xhigh |
| CAISI evaluation of DeepSeek V4 Pro: PUMaC 2024 nist-caisi-deepseek-v4-pro | CAISI/NIST 'CAISI Evaluation of DeepSeek V4 Pro' (page of 1 May 2026, updated 2 May; evaluation April 2026): PUMaC 2024, accuracy % of tasks solved; agentic runs with Inspect ReAct agent (1M weighted-token budget for PortBench/CTF, 500k for SWE-Bench Verified); CAISI-served DeepSeek V4 on H200/B200 GPUs with developer-recommended settings incl. maximum thinking; cost (7 benchmarks) = Figure 4 average expense per solved item over the items BOTH GPT-5.4 mini and DeepSeek V4 Pro solved; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4 mini·xhigh |
| CAISI evaluation of DeepSeek V4 Pro: SMT 2025 nist-caisi-deepseek-v4-pro | CAISI/NIST 'CAISI Evaluation of DeepSeek V4 Pro' (page of 1 May 2026, updated 2 May; evaluation April 2026): SMT 2025, accuracy % of tasks solved; agentic runs with Inspect ReAct agent (1M weighted-token budget for PortBench/CTF, 500k for SWE-Bench Verified); CAISI-served DeepSeek V4 on H200/B200 GPUs with developer-recommended settings incl. maximum thinking; cost (7 benchmarks) = Figure 4 average expense per solved item over the items BOTH GPT-5.4 mini and DeepSeek V4 Pro solved; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4 mini·xhigh |
| CalorieBench github-caloriebench | Anthropic API · 200 food photos, effort=high sent without thinking → nothink regime, within-20 % score ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5.5·high · Claude Sonnet 5·high · GPT-6 Luna·high · GPT-6 Sol·high · GPT-6 Astra·high |
| caloriebench-mae github-caloriebench | — | xmodel | Claude Fable 5.1·high · Claude Opus 5.5·high · Claude Sonnet 5·high · GPT-6 Luna·high · GPT-6 Sol·high · GPT-6 Astra·high |
| Capture-the-Flag Challenges (internal) (OpenAI GPT-5.6 post chart) openai | Capture-the-Flag Challenges (internal), code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| cauchye_asia Japanese text + vision, basic set (21 q x 2) zenn-cauchye_asia | paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·high · Gemini 3.8 Flash·medium · GPT-6 Luna·medium · GPT-6 Sol·medium |
| cauchye_asia Japanese text + vision, hard set (10 q x 2) zenn-cauchye_asia | paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·high · Gemini 3.8 Flash·medium · GPT-6 Luna·medium · GPT-6 Sol·medium |
| CausalGame Agent mode (structured tool calling): mean survival rate over 14 experiments causalgame-2607.04293 | CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Agent mode (tool calls); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓ | xmodel | GLM-5.2·high · GPT-5.5·high · GPT-5.5·medium · GPT-5.5·xhigh · Gemini 3.5 Flash·medium · GPT-5.2·high · Grok 4.20·think · Gemini 3.1 Flash-Lite·minimal |
| CausalGame Prompting mode (Python code): mean survival rate over 14 experiments causalgame-2607.04293 | CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Prompting mode (Python code); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓ | xmodel | GLM-5.2·high · GPT-5.5·high · GPT-5.5·medium · GPT-5.5·xhigh · Gemini 3.5 Flash·medium · GPT-5.2·high · Grok 4.20·think · Gemini 3.1 Flash-Lite·minimal |
| CEO-Bench arxiv-2606.18543 | terminal-agent · Opus/Sonnet=MAX, Haiku=thinking ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · Gemini 3.5 Flash·high · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Sonnet 4.6·high · GPT-5.5·xhigh |
| cfg-bash computingforgeeks | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·nothink |
| cfg-k8s computingforgeeks | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·nothink |
| cfg-terraform computingforgeeks | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·nothink |
| ChaosCypher context-extraction: carrier suite (N=21) github-chaoscypher | War & Peace fixture, mcp:claude-code harness, carrier suite, pass/fail verdicts; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high |
| ChaosCypher context-extraction: chat suite (N=80) github-chaoscypher | War & Peace fixture, mcp:claude-code harness, chat suite, pass/fail verdicts; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Sonnet 5·high |
| ChaosCypher context-extraction: native suite (N=65) github-chaoscypher | War & Peace fixture, mcp:claude-code harness, native suite, pass/fail verdicts; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Sonnet 5·high |
| Chartography (Surge AI) launch snapshot July 2026: cells absent from or different in the held board chartography-2608.10677 | Chartography launch leaderboard (arXiv 2608.10677, frozen July 2026 snapshot; 100 tasks x 20 trials, no tools, mean pass@1 % with 95% CI, Surge-hosted): only the cells that are NOT identical to the held surgehq.ai board (read 2026-10-15, group surge-chartography) are kept; join with that group for the max/xhigh peers; arXiv 2608.10677 (Aug 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·max · Claude Fable 5·max · GPT-5.6 Terra·max · GPT-5.6 Luna·max · GPT-5.5·xhigh · GPT-5.4·xhigh · Grok 4.5·high · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Opus 4.8·max · Grok 4.3·high · Claude Opus 4.8·nothink |
| Chess puzzle accuracy, base prompt (imec APO paper) imec-2610.00416 | 559-puzzle test set, base prompt (no optimisation), mean of 3 runs, generative single-move answer, legal-move list given, no tools; metric puzzle accuracy %; cost = USD per 1,000 solver-position calls; arXiv 2610.00416 Table 2 (Oct 2026); read 2026-10-20 ✓ | xmodel | Gemini 3.5 Flash·medium · Gemini 3.5 Flash-Lite·low · Claude Haiku 4.5·nothink · GPT-5.6 Luna·low |
| ChronoBay watch reference identification from one photo (QutaibaAbualrob/chronobay_ai_bench) QutaibaAbualrob/chronobay_ai_bench | REPORT.md 2026-09-30, 100 images, 1 pass/image, same prompt (sha 4e521b10...); score is exact-reference-number match %; cost is per-image USD computed by the repo from its own token counts and a pinned price table (list-price-cost); read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Haiku 4.5·nothink |
| CL-Bench clbench.com | CL-Bench, learning from context · overall score on 0–1 with its std in the flags; Opus 4.5 with and without thinking as default/nothink; no cost; read 2026-09-30 ✓ | xmodel | GPT-5.4·xhigh · GPT-5.1·high · Grok 4.20·think · Gemini 3.1 Pro·high · GPT-5.2·high · Grok 4.20·nothink · Kimi K2 Thinking·solo · Gemini 3 Pro·high |
| Claude Code Camp 12-task claudecodecamp | beehiiv newsletter · Claude Code, default effort, 12 tasks×3 reps, cost+tokens only ✓ | xmodel | Claude Opus 5.5·medium · GPT-6 Sol·medium |
| claude-maintain-rails-evals: planner variant v12 (release notes), Claude headless, 5 runs/config github-claude-maintain-rails-evals | claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·high |
| claude-maintain-rails-evals: planner variant v16 (Argon clean-room procedure), Claude headless, 5 runs/config github-claude-maintain-rails-evals | claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·high |
| claude-maintain-rails-evals: risk-report variant v20 (changes-first), Opus 5.5 high vs low, 5 runs/config github-claude-maintain-rails-evals | claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·low |
| claude.dev hillclimb anthropic | claude.dev hillclimb post · 30 support tickets, audited prompt, Opus 5.5 vs Sonnet 5 at low, cost per ticket ✓ | xmodel | Claude Opus 5.5·low · Claude Sonnet 5·low |
| ClawProBench closed dataset 68 tasks (OpenClaw) clawprobench-board | ClawProBench (suyoumo.github.io/bench), closed dataset 68 tasks, OpenClaw agent runtime v3.2.6, deterministic grading, repeated trials; Final Score 0-100 (page default rank signal); cost = page cost_usd = (in + 0.5 x (cacheread + cachewrite)) x price_in + out x price_out; rows updated 2026-02..2026-07 (live board); one route kept per couple (earliest updated); read 2026-10-20 ✓ | xmodel | GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.5·xhigh |
| ClawProBench open dataset 102 tasks (OpenClaw) clawprobench-board | ClawProBench (suyoumo.github.io/bench), open dataset 102 tasks, OpenClaw agent runtime v3.2.6, deterministic grading, repeated trials; Final Score 0-100 (page default rank signal); cost = page cost_usd = (in + 0.5 x (cacheread + cachewrite)) x price_in + out x price_out; rows updated 2026-02..2026-07 (live board); one route kept per couple (earliest updated); read 2026-10-20 ✓ | xmodel | GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.5·xhigh · Kimi K2 Thinking·solo |
| ClerkBench banded_meter_billing (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload zenodo-23048151 | Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Opus 5·high · GPT-6 Luna·high · GPT-6 Sol·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · GPT-5.5·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · GPT-6 Astra·high |
| ClerkBench fee_ledger_dropzero (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload zenodo-23048151 | Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Opus 5·high · GPT-6 Luna·high · GPT-6 Sol·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · GPT-5.5·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · GPT-6 Astra·high |
| clerkbench-ch6 zenodo-23048151 | — | xmodel | GPT-5.5·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·medium · GPT-6 Sol·high |
| cliphou.se Motion Graphics Benchmark v1.1: cost per run (Remotion videos) cliphou.se | 3 briefs x 2 runs via OpenRouter, reasoning high; cost = mean OpenRouter generation cost per run over the 6 runs (computed by us); no quality score published (human review pending); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · GPT-6.1 Sol·high · GPT-6 Luna·high · Gemini 3.8 Flash·high · DeepSeek V4.1 Flash·high · Kimi K3·high |
| Code-as-Policy / RoboDojo (2609.39018) — 5 simulated manipulation tasks, +URAI interface arxiv-2609.39018 | Table 1, avg success rate % over 5 tasks x 5 episodes, output tokens k incl. reasoning; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Claude Fable 5.1·xhigh · Claude Opus 5.5·xhigh · GPT-6 Astra·medium |
| Code-as-Policy / RoboDojo (2609.39018) — 5 simulated manipulation tasks, native interface arxiv-2609.39018 | Table 1, avg success rate % over 5 tasks x 5 episodes, output tokens k incl. reasoning; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Claude Fable 5.1·xhigh · Claude Opus 5.5·xhigh · GPT-6 Astra·medium |
| code-review-bench · /ce-code-review github-codereviewbench | kamui/code-review-bench 23716f7 · 12 OSS PRs × 3 trials, same skill snapshot, Claude Code 2.1.284 high, recall and cohort cost ✓ | xmodel | Claude Sonnet 5.5·high · Claude Opus 5.5·high · GPT-6 Luna·high |
| code-review-bench · /thermo review github-codereviewbench | kamui/code-review-bench 23716f7 · 12 OSS PRs × 3 trials, same skill snapshot, Claude Code 2.1.284 high, recall and cohort cost ✓ | xmodel | Claude Sonnet 5.5·high · Claude Opus 5.5·high · GPT-6 Luna·high |
| code-review-bench: Claude 5.5 as claim labeller (12 batches, 493 claims), agreement with GPT-6.1 Sol labels github-codereviewbench | kamui/code-review-bench grader-model-test-2026-10-12; 12 sessions (12 batches of 53 reviews; 493 claims); Claude Code high; score = label agreement with the saved GPT-6.1 Sol labelling (not accuracy); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high |
| Code4Scene private set (95 cases), overall S_model arxiv-2609.36777 | Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; private set (95 cases), overall S_model; read 2026-10-19 ✓ | xmodel | Gemini 3.8 Flash·high · GPT-5.6 Sol·high · GLM-5.3-Flash·max · DeepSeek V4.1 Flash·high |
| Code4Scene public set (95 cases), overall S_model arxiv-2609.36777 | Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; public set (95 cases), overall S_model; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·max · Gemini 3.8 Flash·high · Claude Fable 5.1·max · Claude Opus 5·max · GPT-5.6 Sol·high · GLM-5.3-Flash·max · Grok 4.6·high · DeepSeek V4.1 Flash·high |
| CodeActionBench, Reference harness (single code-generation loop) arxiv-2609.33807 | CodeActionBench (arXiv 2609.33807, 27 Sep 2026): 25 tasks x 3 seeds = 75 attempts per configuration, agentic Code-as-Policy for embodied manipulation; 'Reference' harness for all rows of this group; score = Success (/75 attempts); task coverage /25 is another summary; cost = Table 11 total API-equivalent USD over the 75 attempts (computed by the authors from tokens at list rates); read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · Gemini 3.6 Flash·high · GPT-5.6 Sol·high · Grok 4.6·high · Claude Sonnet 5·high |
| CodeActionBench, vendor harnesses (Codex CLI / Claude Code) arxiv-2609.33807 | CodeActionBench: same 75 attempts; vendor harness rows (GPT-6 Astra in Codex CLI, Claude Opus 5 in Claude Code); Success (/75); cost = Table 11 total API-equivalent USD; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·high · Claude Opus 5·high |
| CodeAgentBench SWE-Bench-Pro, agent Codex CLI 0.135.0 codeagentbench-swepro | CodeAgentBench (suyoumo.github.io/code-agent-bench) SWE-Bench-Pro 151 tasks x 3 attempts = 453 scoreable attempts, agent Codex CLI 0.135.0 (codex-cli 0.135.0), Final Score 0-100 (weights Pass^3 most, page text); no cost on page; page 'Exported 2026-06-18', rows exported 2026-05-22..2026-06-16; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.3-Codex·xhigh |
| CodeRabbit review (Fable 5.1) coderabbit | CodeRabbit blog 1 Sep · 45 tasks/105 issues, Fable 5.1 low/high + Fable 5, Opus 5, score only ✓ | xmodel | Claude Fable 5.1·low · Claude Fable 5.1·high |
| Codex Radar intelligence-efficiency board, Codex CLI harness codexradar | Codex Radar recurring repository-level coding eval (per-task pass over valid tasks; payload mode equal_latest_3: equal-weight latest three valid samples per task); harness codex; snapshot payload stamped 2026-10-01T23:00:27+08:00 read 2026-10-20; score=passed/valid_tasks (derived; payload gives passed, valid_tasks, iq=150*pass_rate); cost=average_price_usd per run (estimated USD-equivalent from token prices); valid_tasks differ per row (36..336); read 2026-10-20 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.5·high · GPT-5.5·xhigh · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash·high · Grok 4.6·high · Grok 4.6·medium · Grok 4.6·low · Grok 4.6·xhigh · Grok 4.7·high · Grok 4.7·medium · Grok 4.7·low · Grok 4.7·xhigh · GLM-5.3·low · GLM-5.3·high · GLM-5.3·max · GLM-5.3-Flash·low · GLM-5.3-Flash·high · GLM-5.3-Flash·max · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max |
| computingforgeeks: Claude Code bug-fix task (log parser) computingforgeeks | 4-bug Python log parser + 8-test pytest suite, headless Claude Code, 3 runs/model, default effort per model; score saturated (9/9 green) -> cost only; read 2026-10-01 ✓ | xmodel | Claude Sonnet 5·high · Claude Opus 5.5·medium |
| Conceptual Reasoning Index · DTBench conceptualreasoning.ai | Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·xhigh · Claude Opus 5.5·medium · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5·xhigh · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · Claude Opus 5·high · Claude Sonnet 5.5·medium · GPT-6 Astra·xhigh · Claude Fable 5.1·high · Claude Sonnet 5.5·high · GPT-6 Astra·max · Claude Opus 5·max · GPT-6 Astra·high · Claude Sonnet 5.5·low · GPT-6 Astra·medium · GPT-6.1 Sol·max · Claude Fable 5·max · Gemini 4 Argon·low · Claude Opus 5·medium · Gemini 4 Argon·high · Claude Fable 5.1·low · Claude Fable 5·xhigh · Gemini 4 Argon·medium · Claude Opus 5·low · GPT-6 Astra·low · Claude Fable 5·high · Claude Fable 5·medium · GPT-6 Sol·max · GPT-5.6 Sol·max · Claude Fable 5·low · GPT-5.6 Sol·xhigh · Claude Opus 4.8·max · GPT-5.6 Sol·high · GPT-5.5·xhigh · GPT-5.6 Sol·medium · GPT-5.6 Sol·low · GPT-5.5 Pro·xhigh · Gemini 3.8 Flash·medium · GPT-5.6 Terra·max · Claude Opus 4.7·max · Gemini 3.1 Pro·medium · GPT-5.4·xhigh · Claude Haiku 5.5·max · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.1 Pro·high · GLM-5.3·max · Grok 4.7·xhigh · Gemini 3.1 Pro·low · Gemini 3.7 Flash·medium · GPT-5.6 Terra·xhigh · Kimi K3·max · Grok 4.6·xhigh · Claude Sonnet 5·max · GPT-5.6 Terra·high · Claude Sonnet 5·xhigh · GPT-5.6 Terra·medium · Grok 4.5·high · GPT-5.6 Terra·low · Gemini 3.7 Flash·low · Gemini 3.5 Flash·high · Claude Haiku 5.5·xhigh · Claude Sonnet 5·high · Gemini 3.6 Flash·high · DeepSeek V4 Pro 0813 build·max · Claude Sonnet 4.6·high · GPT-5.2·xhigh · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · Claude Haiku 5.5·high · DeepSeek V4.1 Flash·max · GPT-6 Luna·max · GPT-5.6 Luna·high · Claude Haiku 5.5·medium · GPT-5.1·high · Claude Sonnet 5·medium · Claude Haiku 5.5·low · Gemini 3 Flash·high · Qwen3.8-27B·xhigh · Claude Sonnet 5·low · GPT-5.6 Luna·medium · Grok 4.3·high · GPT-5.6 Luna·low · GPT-5.4 mini·xhigh · GPT-5.2 Chat (latest)·solo · Gemini 3.5 Flash-Lite·high · GPT-5.4 nano·xhigh · Gemini 3.1 Flash-Lite·high |
| Conceptual Reasoning Index · LMCA conceptualreasoning.ai | Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·xhigh · Claude Opus 5.5·medium · Claude Fable 5.1·medium · Claude Fable 5.1·max · Claude Opus 5·xhigh · Claude Opus 5.5·low · Claude Fable 5.1·xhigh · Claude Opus 5·high · Claude Sonnet 5.5·medium · GPT-6 Astra·xhigh · Claude Fable 5.1·high · Claude Sonnet 5.5·high · GPT-6 Astra·max · Claude Opus 5·max · GPT-6 Astra·high · Claude Sonnet 5.5·low · GPT-6 Astra·medium · GPT-6.1 Sol·max · Claude Fable 5·max · Gemini 4 Argon·low · Claude Opus 5·medium · Gemini 4 Argon·high · Claude Fable 5.1·low · Claude Fable 5·xhigh · Gemini 4 Argon·medium · Claude Opus 5·low · GPT-6 Astra·low · Claude Fable 5·high · Claude Fable 5·medium · GPT-6 Sol·max · GPT-5.6 Sol·max · Claude Fable 5·low · GPT-5.6 Sol·xhigh · Claude Opus 4.8·max · GPT-5.6 Sol·high · GPT-5.5·xhigh · GPT-5.6 Sol·medium · GPT-5.6 Sol·low · GPT-5.5 Pro·xhigh · Gemini 3.8 Flash·medium · GPT-5.6 Terra·max · Claude Opus 4.7·max · Gemini 3.1 Pro·medium · GPT-5.4·xhigh · Claude Haiku 5.5·max · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.1 Pro·high · GLM-5.3·max · Grok 4.7·xhigh · Gemini 3.1 Pro·low · Gemini 3.7 Flash·medium · GPT-5.6 Terra·xhigh · Kimi K3·max · Grok 4.6·xhigh · Claude Sonnet 5·max · GPT-5.6 Terra·high · Claude Sonnet 5·xhigh · GPT-5.6 Terra·medium · Grok 4.5·high · GPT-5.6 Terra·low · Gemini 3.7 Flash·low · Gemini 3.5 Flash·high · Claude Haiku 5.5·xhigh · Claude Sonnet 5·high · Gemini 3.6 Flash·high · DeepSeek V4 Pro 0813 build·max · Claude Sonnet 4.6·high · GPT-5.2·xhigh · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · Claude Haiku 5.5·high · DeepSeek V4.1 Flash·max · GPT-6 Luna·max · GPT-5.6 Luna·high · Claude Haiku 5.5·medium · GPT-5.1·high · Claude Sonnet 5·medium · Claude Haiku 5.5·low · Gemini 3 Flash·high · Qwen3.8-27B·xhigh · Claude Sonnet 5·low · GPT-5.6 Luna·medium · Grok 4.3·high · GPT-5.6 Luna·low · GPT-5.4 mini·xhigh · GPT-5.2 Chat (latest)·solo · Gemini 3.5 Flash-Lite·high · GPT-5.4 nano·xhigh · Gemini 3.1 Flash-Lite·high |
| contextarena-needle8-auc128k contextarena.ai | — | xmodel | Qwen3.8 Max·low · Qwen3.8 Max·xhigh |
| Convex evals (no guidelines, web research) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines, web research' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·medium · Claude Opus 5·medium · Claude Sonnet 5.5·medium · Claude Opus 4.8·medium · Claude Sonnet 5·medium · GPT-6 Astra·medium · GPT-6.1 Sol·medium · GPT-5.6 Sol·medium · GPT-5.5·medium · Grok 4.6·medium · Grok 4.5·medium · GPT-6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Luna·medium · GPT-5.6 Luna·medium · DeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high · GLM-5.2·high |
| Convex evals (no guidelines) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5·medium · Claude Sonnet 5.5·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Fable 5.1·medium · Claude Sonnet 5·medium · GPT-6 Astra·medium · GPT-6.1 Sol·medium · GPT-5.6 Sol·medium · Grok 4.6·medium · Grok 4.7·medium · Grok 4.5·medium · GPT-6 Sol·medium · GPT-5.5·medium · GPT-5.6 Terra·medium · GPT-6 Luna·medium · Gemini 3.8 Flash·medium · GPT-5.6 Luna·medium · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high · GLM-5.2·high |
| Convex evals (with guidelines) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'with guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Claude Sonnet 5.5·medium · Claude Fable 5·medium · Claude Fable 5.1·medium · Claude Opus 5.5·medium · Claude Opus 5·medium · Claude Opus 4.8·medium · Claude Sonnet 5·medium · Grok 4.6·medium · GPT-6 Astra·medium · GPT-6.1 Sol·medium · Grok 4.7·medium · GPT-5.6 Sol·medium · GPT-6 Sol·medium · Grok 4.5·medium · GPT-5.5·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · GPT-6 Luna·medium · Gemini 3.8 Flash·medium · GLM-5.2·high · DeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high |
| Convex LLM leaderboard v92a60e0b · no guidelines convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines; read 2026-10-15 ✓ | xmodel | GPT-6 Astra·medium · Claude Opus 5.5·medium · GPT-6.1 Sol·medium · Claude Fable 5·medium · Claude Sonnet 5.5·medium · GPT-5.6 Sol·medium · Claude Fable 5.1·medium · Grok 4.6·medium · Claude Opus 5·medium · Grok 4.7·medium · Claude Opus 4.8·medium · Grok 4.5·medium · GPT-5.5·medium · Claude Sonnet 5·medium · GPT-5.6 Terra·medium · GPT-6 Luna·medium · Gemini 3.8 Flash·medium · GPT-5.6 Luna·medium · DeepSeek V4 Flash preview·high · GLM-5.2·high · DeepSeek V4 Pro preview·high · Gemini 4 Argon·medium · Claude Haiku 5.5·medium |
| Convex LLM leaderboard v92a60e0b · no guidelines + web convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines + web; read 2026-10-15 ✓ | xmodel | Gemini 4 Argon·medium · Claude Opus 5.5·medium · GPT-6.1 Sol·medium · GPT-6 Astra·medium · Claude Fable 5·medium · Claude Fable 5.1·medium · GPT-5.6 Sol·medium · Claude Sonnet 5.5·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Sonnet 5·medium · Grok 4.5·medium · GPT-5.6 Terra·medium · GPT-6 Luna·medium · GPT-5.6 Luna·medium · GLM-5.2·high · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high · Claude Haiku 5.5·medium |
| Convex LLM leaderboard v92a60e0b · with guidelines convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, with guidelines; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Astra·medium · Claude Fable 5·medium · Claude Fable 5.1·medium · GPT-5.6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.5·medium · Grok 4.6·medium · Grok 4.7·medium · Claude Opus 5·medium · Claude Opus 4.8·medium · GPT-5.5·medium · GPT-5.6 Terra·medium · Claude Sonnet 5·medium · GPT-5.6 Luna·medium · GPT-6 Luna·medium · Gemini 3.8 Flash·medium · GLM-5.2·high · DeepSeek V4 Pro preview·high · Gemini 4 Argon·medium · Claude Haiku 5.5·medium |
| costrouter github-costrouter | — | xmodel | Claude Opus 5·low |
| CRBugBench post-merge bugs (Fix track), recall@5 arxiv-2610.13668 | 500 review tasks per track from 98 OSS repos, copilot-cli 1.0.74 official /review prompt, default prompt; Table 1 of arXiv 2610.13668; read 2026-10-15 ✓ | xmodel | Claude Opus 5·xhigh · Claude Opus 4.8·xhigh · GPT-5.6 Sol·xhigh · GPT-5.5·xhigh · GPT-5.4·xhigh · Claude Sonnet 5·xhigh · GPT-5.6 Luna·xhigh · Claude Haiku 4.5·nothink |
| CRBugBench pre-merge bugs (Cmt track), recall@5 arxiv-2610.13668 | 500 review tasks per track from 98 OSS repos, copilot-cli 1.0.74 official /review prompt, default prompt; Table 1 of arXiv 2610.13668; read 2026-10-15 ✓ | xmodel | Claude Opus 5·xhigh · Claude Opus 4.8·xhigh · GPT-5.6 Sol·xhigh · GPT-5.5·xhigh · GPT-5.4·xhigh · Claude Sonnet 5·xhigh · GPT-5.6 Luna·xhigh · Claude Haiku 4.5·nothink |
| ctala github-ctala | Claude Code CLI · reasoning not configured, temp 0.7 ✓ | xmodel | GPT-6 Luna Pro·medium · Claude Haiku 5.5·nothink · Claude Sonnet 5.5·high · Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-5.6 Luna·medium · DeepSeek V4.1 Flash·high · GLM-5.3-Flash·max · Gemini 3.5 Flash-Lite·minimal · GPT-6 Luna·medium · Gemini 3.6 Flash·medium · Qwen3.8 Max·xhigh · Gemini 3.7 Flash·medium · GLM-5.2·high · GPT-5.6 Terra·medium · Gemini 3.8 Flash·medium · GPT-5.6 Sol·medium · Grok 4.6·high · Grok 4.5·high · DeepSeek V4 Flash 0731 build·high · Kimi K2.7 Code·solo · Kimi K2 Thinking·solo · Claude Opus 5·high · Claude Haiku 4.5·nothink |
| CTI-REALM unconstrained (300-message budget): overall reward ctirealm-2610.14180 | Unconstrained CTI-REALM (Averlon), Table 2: 50 tasks, one sample each, common harness, judges Gemma-4-31B thinking / DeepSeek-v4-flash / GLM-5.3-flash x3 trials, message budget 300 (14 models have samples run at 300 incl. 10 spliced, 12 carried from lower budgets with zero capped samples); overall weighted reward; checkpoints C0-C4 same runs (rule 8); cost = cost of the 50-task run at list prices; arXiv 2610.14180 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·high · Claude Opus 5·high · DeepSeek V4.1 Flash·high · Claude Haiku 4.5·nothink |
| CUA-SWE mobile hybrid: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, mobile domain (20 tasks), deterministic executable verifiers, access = hybrid CUA (screenshots + graphical actions); frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·high · GPT-5.6 Sol·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Opus 4.8·high · Claude Sonnet 5·high · Claude Fable 5·high · Grok 4.6·high |
| CWE-bench (120 vulnerability-repair tasks, 4 attempts) arxiv-2610.17020 | Claude Code, high reasoning effort, 4 attempts per task, 1 h limit; programmatic pass@1; cost = mean API cost per rollout; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · Gemini 4 Argon·high · GPT-6 Astra·high · GPT-6 Sol·high |
| cwe-bench-v2-class-k01 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k02 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k03 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k04 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k05 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k06 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k07 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k08 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k09 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-class-k10 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-combined-pass1 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-combined-pass4 cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-language-c-cpp cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-language-go cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-language-java cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-language-js-ts cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-language-python cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| cwe-bench-v2-language-rust cwe-bench | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·high |
| CyberGym (table) openai | CyberGym; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Data Science Tasks (Internal) (OpenAI GPT-6 Astra post chart) openai | Data Science Tasks (Internal); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · Claude Fable 5.1·max · Claude Opus 5·max · Claude Fable 5·max |
| DataLLM Lab: 9 executed Python tasks, Qwen3.8 Max family (Prime, Max, Max-0902), pass and cost per 1,000 tasks datallmlab | 9 executed Python tasks (python3 -I subprocess, 12 s timeout), temperature 0, max_tokens 4000, one scored attempt, via OpenRouter; three runs on different days (Prime 2026-10-02, Max 2026-08-20, Max-0902 2026-09-16); cost = measured tokens x list price on the priced-on date (source-computed, 'not a billing statement'); read 2026-10-20 ✓ | xmodel | Gemini 3.6 Flash·medium · Grok 4.5·high · Gemini 3.1 Pro·medium · Claude Haiku 4.5·nothink · GPT-5.6 Sol·medium · Claude Sonnet 4.6·nothink · GPT-5.4·nothink · Claude Opus 5·high · Kimi K3·max · Gemini 3.5 Flash·medium · GLM-5.1·think · GLM-5·think · GPT-5.4 mini·nothink · Claude Opus 4.7·nothink · GLM-5.3·max · Gemini 3.7 Flash·medium · Grok 4.6·high · Grok 4.20·nothink · GPT-6 Astra·medium · Claude Fable 5.1·high · Gemini 3.8 Flash·medium · DeepSeek V4.1 Flash·high · Qwen3.8 Max 0902 build·xhigh · GPT-6 Sol·medium · GPT-6 Luna·medium · GPT-6 Luna Pro·medium · GPT-6.1 Sol·medium · Claude Opus 5.5·high · Grok 4.7·high · GLM-5.3-FlashX·max |
| datallm-effort-probe datallmlab | — | xmodel | GPT-6 Luna·medium · GPT-6.1 Sol·medium · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high |
| datallmlab-coding9 datallmlab | — | xmodel | GLM-5.2·high · Kimi K2.7 Code·solo |
| Dealwatch defect review github-dealwatch | Claude Code orchestrator · 5 re-seeded bugs, Fable 5.1 vs Opus 5.5 @high, 1 run, list-price cost ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5.5·high |
| Dealwatch small feature github-dealwatch | same repo · 6-req feature, hidden tests saturated → cost only ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5.5·high |
| decisionpivot-core-pair arxiv-2610.28522 | — | xmodel | GLM-5.3·low |
| decisionpivot-subset1296 arxiv-2610.28522 | — | xmodel | GLM-5.3·low · GLM-5.3·high |
| DeepSeek V4 preview card: apex deepseek | apex: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: apexshort deepseek | apexshort: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: browsecomp deepseek | browsecomp: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: codeforces deepseek | codeforces: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: corpusqa deepseek | corpusqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: csimpleqa deepseek | csimpleqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: gdpval deepseek | gdpval: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think |
| DeepSeek V4 preview card: gpqa deepseek | gpqa: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: hle deepseek | hle: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: hle-tools deepseek | hle-tools: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: hmmt26 deepseek | hmmt26: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: imo deepseek | imo: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: lcb deepseek | lcb: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: mcpatlas deepseek | mcpatlas: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: mmlupro deepseek | mmlupro: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: mrcr deepseek | mrcr: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: simpleqav deepseek | simpleqav: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: swemulti deepseek | swemulti: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: swepro deepseek | swepro: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: sweverified deepseek | sweverified: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: tb20 deepseek | tb20: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek V4 preview card: toolathlon deepseek | toolathlon: DeepSeek-V4 preview model card (HF README, 2026-04-22/24); n, attempts, scaffold, judge not stated in the card (no footnote on sampling); DeepSeek rows by reasoning mode (Non-think/Think High/Think Max), other makers per column header; Table 1 (frontier) and Table 2 (modes) merged, the V4-Pro Max cell is identical in both; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · Gemini 3.1 Pro·high · GLM-5.1·think · DeepSeek V4 Flash preview·high · DeepSeek V4 Pro preview·high |
| DeepSeek-V3.2 report: aime deepseek | aime; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: browsecomp-cm deepseek | BrowseComp with the context-management technique (marked *), search agent with a commercial search API; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: browsecompzh deepseek | browsecompzh; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: gpqa deepseek | gpqa; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hle deepseek | hle; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hle-search deepseek | hle-search; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hmmt25 deepseek | hmmt25; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hmmtnov deepseek | hmmtnov; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: imo deepseek | imo; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: lcb deepseek | lcb; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: mcpmark deepseek | mcpmark; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: mcpuniverse deepseek | mcpuniverse; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: mmlupro deepseek | mmlupro; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: swemulti deepseek | swemulti; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: sweverified deepseek | sweverified; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: tau2 deepseek | tau2; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: tooldec deepseek | tooldec; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| deepseek-v4-flash-0731-card: deepswe deepseek | DeepSWE (v1.1 per later cards); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | DeepSeek V4 Pro 0813 build·max |
| deepseek-v4-flash-0731-card: nl2repo deepseek | NL2Repo; DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | DeepSeek V4 Pro 0813 build·max |
| deepseek-v4-flash-0731-card: tb21 deepseek | Terminal-Bench 2.1 (Pass@1); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | DeepSeek V4 Pro 0813 build·max |
| deepseek-v41-techreport: ale deepseek | Agents' Last Exam (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold(ALE-CLI); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: autobench deepseek | AutomationBench v1.0.6 public set (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: babyvision deepseek | BabyVision w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max |
| deepseek-v41-techreport: chartography deepseek | Chartography w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max |
| deepseek-v41-techreport: cybergym deepseek | CyberGym (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: exploitgym deepseek | ExploitGym (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: hle deepseek | HLE (Pass@1), full set; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: hle-tools deepseek | HLE with tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: nl2repo deepseek | NL2Repo-Bench (Score); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: programbench deepseek | ProgramBench (Almost@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: tb30 deepseek | Terminal-Bench 3.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: tb40 deepseek | Terminal-Bench 4.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max · GLM-5.3·max · DeepSeek V4.1 Flash·max |
| deepseek-v41-techreport: zerobench deepseek | ZeroBench-main w/ tools (Pass@5); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Claude Opus 5·max · GPT-5.6 Sol·max · Kimi K3·max |
| DeepSWE (OpenAI GPT-6 Sol and Luna post) openai | DeepSWE; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| DeepSWE (OpenAI GPT-6.1 Sol post) openai | DeepSWE; OpenAI GPT-6.1 Sol post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; Wayback first capture; read 2026-10-19 ✓ | xmodel | GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max |
| DeepSWE 1.0 (xAI Grok 4.5 post) xai | DeepSWE 1.0 pass@1 %; footnote Eval created by Datacurve, run with each model provider's harnesses by AA; xAI post 16 Jul 2026; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · Claude Opus 4.8·max · Claude Opus 4.7·max |
| DeepSWE v1.1 (OpenAI GPT-5.6 post chart) openai | DeepSWE v1.1 pass@1, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Gemini 3.1 Pro·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max |
| DeepSWE v1.1 (OpenAI GPT-6 Astra post chart, non-code mode) openai | DeepSWE v1.1; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high |
| DeepSWE v1.1 (xAI Grok 4.6 post) xai | DeepSWE v1.1 pass@1 %, mini-swe-agent harness run by Datacurve; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · GPT-5.6 Sol·max |
| DeepSWE v1.1 (xAI Grok 4.7 post) xai | DeepSWE v1.1, resolve rate %, harness/n/attempts not stated; xAI Grok 4.7 launch post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·high · Grok 4.6·high · GPT-5.6 Sol·max · Claude Fable 5.1·max |
| deepu-cc-set1-of8 deepu | — | xmodel | Claude Opus 5.5·xhigh · Claude Opus 5.5·medium |
| deepu-cc-set2-of8 deepu | — | xmodel | Claude Opus 5.5·xhigh · Claude Opus 5.5·medium |
| Deloitte code review anthropic-partner-deloitte | Opus 5.5 launch page partner quote · bugs caught, score only, n unstated ✓ | xmodel | Claude Opus 5.5·low · Claude Opus 5·high |
| dentsu-b dentsusoken | — | xmodel | Claude Sonnet 5·high · Claude Opus 5·high · Claude Fable 5·high |
| dentsu-d dentsusoken | — | xmodel | Claude Sonnet 5·high · Claude Opus 5·high · Claude Fable 5·high |
| Design Arena - all categories designarena | designarena.ai/leaderboard 'allcategories': crowd human-vote Elo on AI-generated designs/UI (1,817,484 votes, 180 models); effort not in the Claude slugs (other vendors' slugs carry -max/-high) so default; no cost; read 2026-10-15 ✓ | xmodel | GPT-6 Astra·max · GPT-5.6 Sol·xhigh · Kimi K2.7 Code·solo · Grok 4.20·think · Grok 4.20·nothink · GPT-5.4·low · GPT-5.4·nothink · GPT-5.1·high · GPT-5.1·medium · GPT-5.1·low |
| Design Arena - fullstack designarena | designarena.ai/leaderboard 'fullstack' category: crowd human-vote Elo on full-stack app generation (61,252 votes, 49 models); effort not stated; no cost; read 2026-10-15 ✓ | xmodel | GPT-5.6 Sol·xhigh · Kimi K2.7 Code·solo · Grok 4.20·think |
| Design Creativity Bench (UI generation; overall T-score, median cost per design) arxiv-2610.11539 | 80 UI design tasks / 160 prompts, one generation per design, highest effort (Opus 5.5 at xhigh); Tables 2 and 9 of arXiv 2610.11539; read 2026-10-15 ✓ | xmodel | GPT-6 Astra·max · GPT-6.1 Sol·max · Grok 4.7·xhigh · Gemini 3.8 Flash·high · GPT-5.6 Sol·max · Claude Opus 5.5·xhigh · Claude Fable 5.1·max · DeepSeek V4.1 Flash·max · Grok 4.6·xhigh · GLM-5.3·max · Kimi K3·max · GLM-5.3-Flash·max |
| Design Tasks (Internal) (OpenAI GPT-6 Astra post chart) openai | Design Tasks (Internal); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · Claude Fable 5.1·max |
| DevelopersIO Project Euler 1012 (hard), Claude 5 family + Haiku 5.5 effort ladder classmethod | claude -p --effort medium (trio) / Haiku 5.5 five efforts x 2 runs; Bash/Read/Write/Edit/WebFetch; cost = Claude Code estimate; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·medium |
| DexHoldem (2605.18727v2) — 36 agentic perception problems, Claude Code, avg of 3 runs arxiv-2605.18727 | Table 2, overall strict % and per-category average column; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | GPT-6 Astra·medium · GPT-6 Sol·medium · GPT-5.6 Sol·medium · GPT-5.6 Luna·medium · GPT-5.5·medium · GPT-5.4 mini·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium |
| dexholdem-overall arxiv-2605.18727 | — | xmodel | GPT-6 Astra·medium · GPT-6 Sol·medium · GPT-5.6 Sol·medium · GPT-5.6 Luna·medium · GPT-5.5·medium · GPT-5.4 mini·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium |
| DFIR-Metric Module I (713 MCQ), mean accuracy (independent safety evaluation of Kimi K2.5) arxiv-2604.03121 | DFIR-Metric, 5 trials at temp 1.0, single prompt no code execution; extended reasoning enabled; paper 2026-04-03; read 2026-10-20 ✓ | xmodel | GPT-5.2 Pro·xhigh · GPT-5.2·xhigh |
| DFIR-Metric Module II (150 CTF), Confidence Index (independent safety evaluation of Kimi K2.5) arxiv-2604.03121 | DFIR-Metric, 5 trials at temp 1.0, single prompt no code execution; extended reasoning enabled; paper 2026-04-03; read 2026-10-20 ✓ | xmodel | GPT-5.2 Pro·xhigh · GPT-5.2·xhigh |
| digirise/チャエン: 3 prompts (game/summary/earnings), total time/cost, n=1 digirise-chaen | digirise.ai/chaen-ai-lab/claude-opus-5-5/ (cites チャエン's X post 2026-09-23); Claude Code, 1 run per model per prompt, 3 prompts summed; effort-from-config: "Claudeの2モデルはClaude Code経由でeffortはxhigh"; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · GPT-6 Astra·high · Gemini 3.8 Flash·high |
| discoverphysics-ara-codex hf-agentnativeresearchlab | — | xmodel | GPT-5.6 Sol·high · GPT-5.5·high |
| Double-pendulum job, Haiku 5.5 effort ladder + Sonnet 5.5 medium (AI Receipts video) youtube-aireceipts | 3 runs each; pass = energy drift < 0.1% at 2 simulated min; cost = exact token counts x list price; 2026-10-08; read 2026-10-18 ✓ | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·max |
| DrivingBench: progress along a 127 m real-car cone course (best attempt, %) drivingbench-2609.38948 | DrivingBench Table 2: VLM drives a real Toyota Corolla through 3 tools, one chat session per model = one trial (n=1), up to 3 attempts with retained context and a fixed reflection prompt; progress % = furthest point on the centreline (best attempt); each model in its native harness (Codex CLI 0.154.0-alpha.6.2 for GPT-6 Astra and GPT-5.6 Sol; Claude Code 2.1.274; Cursor); cost = tokens x public list prices as of Sep 2026 summed over all attempts; values typed by hand from the page-7 image (the text layer runs digits together); arXiv 2609.38948 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-6 Astra·medium · Claude Fable 5.1·medium · Grok 4.6·medium · GPT-5.6 Sol·medium |
| Drug-asset scouting completeness benchmark (Bioptic), baseline systems arxiv-2602.15019 | Hunt Globally (arXiv 2602.15019, 16 Feb 2026): asset scouting eval, LLM-as-judge calibrated to experts; score = F1 (Table 2; recall and precision are the same runs); the authors' Bioptic Agent (GPT-5.2 high) is the system under test and is not a row; read 2026-10-20 ✓ | xmodel | GPT-5.2 Pro·high · GPT-5.2·high |
| dubesor-benchtable dubesor | — | xmodel | Grok 4.20·think · GLM-5·think · Kimi K2 Thinking·solo · GLM-4.7·think · GPT-5.1 Chat (latest)·solo · GPT-5.2 Chat (latest)·solo · Grok 4.20·nothink |
| Dyad app-builder · Deskhero github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Deskhero (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | GPT-6.1 Sol·medium · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Sol·medium · GPT-6 Astra·low · GPT-5.6 Sol·medium · Grok 4.6·medium · Claude Fable 5.1·medium · Claude Opus 5·medium · Claude Fable 5·medium · GPT-6 Luna·medium · Grok 4.7·high · Gemini 3.8 Flash·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · Grok 4.5·medium · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Grok 4.6·high · Gemini 3.8 Flash·high · GPT-6 Astra·medium · DeepSeek V4 Flash 0731 build·high |
| Dyad app-builder · Portalis github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Portalis (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | GPT-6.1 Sol·medium · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Sol·medium · GPT-6 Astra·low · GPT-5.6 Sol·medium · Grok 4.6·medium · Claude Fable 5.1·medium · Claude Opus 5·medium · Claude Fable 5·medium · GPT-6 Luna·medium · Grok 4.7·high · Gemini 3.8 Flash·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · Grok 4.5·medium · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Grok 4.6·high · Gemini 3.8 Flash·high · GPT-6 Astra·medium · DeepSeek V4 Flash 0731 build·high |
| Dyad app-builder · Relay CRM github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Relay CRM (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | GPT-6.1 Sol·medium · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Sol·medium · GPT-6 Astra·low · GPT-5.6 Sol·medium · Grok 4.6·medium · Claude Fable 5.1·medium · Claude Opus 5·medium · Claude Fable 5·medium · GPT-6 Luna·medium · Grok 4.7·high · Gemini 3.8 Flash·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · Grok 4.5·medium · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Gemini 3.8 Flash·high · GPT-6 Astra·medium · DeepSeek V4 Flash 0731 build·high |
| E-Bench (MCP tools): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench (MCP tools): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Kimi K3·max · GPT-5.5·xhigh · Claude Opus 4.8·max · Grok 4.5·high · Gemini 3.5 Flash·high |
| E-Bench-Code (code execution): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench-Code (code execution): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Kimi K3·max · GPT-5.5·xhigh · Claude Opus 4.8·max · Grok 4.5·high · Gemini 3.5 Flash·high |
| E-Commerce Bench (arXiv 2608.30730) arxiv-2608.30730 | E-Commerce Bench, simulated e-commerce store, 365-day task, 18 tools, 4000-turn cap, 5 independent episodes per model in one fixed world, harness = the paper's agent loop; score = mean end-of-year total assets (kilo-yuan) from a 100k stake, bankrupt episodes kept in the mean; no cost reported; arXiv v1 2026-08-31; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·max · Claude Fable 5·max · GPT-5.5·xhigh · Claude Opus 4.8·max · Claude Opus 4.7·max · Gemini 3.5 Flash·high · Gemini 3.1 Pro·high · GLM-5.2·high · GLM-5.1·think |
| econ-eval leaderboard A (20 tasks, Opus-5 judge) github-deluair-econ-eval | deluair/econ-eval README at commit 77f9f3e (16 Oct 2026), leaderboard of 2026-09-10: 20 economics tasks (quantitative, reasoning, coding, writing) x 5 samples = 100 completions/model, numeric/exact/code-exec graders + rubric tasks judged by Claude Opus 5; score = mean over tasks of task means; cost = total for the 100 completions from token counts x list price (prices read from the OpenRouter catalog 2026-07-31 / vendor pages); per-track tables are the same completions (not extra groups); ten models ran 2026-08-01, deepseek-flash added 2026-09-10 same tasks/samples/judge; read 2026-10-20 ✓ | xmodel | DeepSeek V4.1 Flash·high · Claude Opus 4.8·high · Kimi K3·max · Gemini 3.6 Flash·medium · GPT-5.6 Luna·medium · Grok 4.5·high · DeepSeek V4 Flash 0731 build·high · GLM-5.2·max |
| econ-eval leaderboard B (50 tasks, GPT-6 Astra judge) github-deluair-econ-eval | deluair/econ-eval README at commit 77f9f3e, leaderboard B: 50 tasks (the 20 above + 30 daily-work tasks added 2026-09-15), graded by GPT-6 Astra; 5 samples per task (Argon 1); runs dated 2026-09-15 / 2026-10-13 / 2026-10-15; cost only where an API price exists; different judge and task set from leaderboard A: separate group; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · DeepSeek V4.1 Flash·high |
| EEBench (xAI Grok 4.7 post) xai | EEBench electrical engineering accuracy %, harness/n not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · GPT-5.6 Sol·max · Claude Fable 5.1·max · GPT-6 Astra·max |
| EMB vals.ai | Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max |
| energy-before-only arxiv-2610.26538 | — | xmodel | Claude Haiku 5.5·low · Claude Sonnet 5.5·low · Claude Opus 5.5·low · GPT-6 Luna·low · GPT-6.1 Sol·low · GPT-6 Astra·low |
| energy-both-photos arxiv-2610.26538 | — | xmodel | Claude Haiku 5.5·low · Claude Sonnet 5.5·low · Claude Opus 5.5·low · GPT-6 Luna·low · GPT-6.1 Sol·low · GPT-6 Astra·low |
| ENSO stochastic-model building (SciExam), final composite score arxiv-2610.10513 | agent builds an ENSO model within 6 h eligible time, one run per system, Claude Code on subscription; Table 4/6 of arXiv 2610.10513; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·medium · Claude Opus 5·medium · GPT-5.6 Sol·medium · GPT-6 Astra·medium · GPT-5.5·medium · Gemini 3.8 Flash·medium |
| Epoch · chess puzzles epoch.ai | Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Qwen3.5-35B-A3B·nothink · Qwen3.6-27B·nothink · Qwen3.7 Flash·nothink · Gemini 3.1 Flash-Lite·minimal · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high · DeepSeek V4.1 Flash 1017 build·max · GPT-5.4 nano·xhigh · GPT-5.2·xhigh · GPT-5.2 Pro·xhigh · GPT-5.4 Pro·xhigh · GPT-5.5 Pro·xhigh · GLM-5.2·high · GLM-5.3·max · Gemini 4 Argon·high · Kimi K2.7 Code·solo · Kimi K3·max · Kimi K3·high · Kimi K3·low · GLM-5.3-Flash·max · GPT-5.2·medium · GPT-5.1·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·nothink · GPT-5.4·xhigh · GPT-5.4·high · GPT-5.4·medium · GPT-5.4·low · GPT-5.4·nothink · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·nothink · GPT-5.4 mini·xhigh · GPT-5.4 mini·high · GPT-5.5·xhigh · GPT-5.4 mini·nothink · GPT-5.5·low · GPT-5.5·nothink · GPT-5.6 Luna·max · GPT-5.6 Luna·low · GPT-5.6 Luna·nothink · GPT-5.6 Terra·max · GPT-5.6 Terra·low · GPT-5.6 Terra·nothink · GPT-5.6 Sol·max · GPT-5.6 Sol·low · GPT-5.6 Sol·nothink · GPT-6 Astra·max · GPT-6 Sol·max · GPT-6 Luna·max · Claude Sonnet 4.6·high · Claude Opus 4.7·max · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.8·max · Claude Opus 4.8·low · Grok 4.20·think · Claude Fable 5·max · Grok 4.3·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·low · Claude Fable 5·high · Claude Fable 5·low · Claude Sonnet 5·max · Claude Sonnet 5·xhigh · Gemini 3.5 Flash-Lite·minimal · Claude Opus 5·max · Gemini 3.6 Flash·high · Grok 4.5·high · Grok 4.6·xhigh · Qwen3.8 Max 0902 build·xhigh · DeepSeek V4 Pro 0813 build·max · Claude Opus 5·low · Claude Fable 5.1·max · Gemini 3.6 Flash·low · Gemini 3.6 Flash·minimal · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Gemini 3.8 Flash·high · Grok 4.6·high · GPT-6.1 Sol·max · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · Grok 4.7·xhigh · Claude Haiku 5.5·low · GPT-5.1·low · Gemini 3 Flash·high · DeepSeek V4 Pro 0813 build·high · Claude Fable 5.2·max · Gemini 4 Carbon·high |
| Epoch · ebr bench epoch.ai | Epoch AI's own evaluation (ebr_bench), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | GPT-6.1 Terra·max · Gemini 4 Argon·high · Claude Haiku 5.5·max · GPT-6.1 Sol·max · Claude Opus 5.5·max · GPT-6 Sol·max · Claude Fable 5.1·max · Claude Opus 5·max · GPT-5.6 Sol·max · GPT-6 Astra·max · Grok 4.6·xhigh · Claude Opus 4.8·max · GPT-5.5·xhigh · Claude Fable 5·max · Claude Opus 4.7·max · GPT-5.2·xhigh · GPT-5.4·xhigh · Claude Fable 5.2·max · Gemini 4 Carbon·high |
| Epoch · frontiermath epoch.ai | Epoch AI's own evaluation (frontiermath), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.4 mini·high · GPT-5.4 nano·high · GPT-5.4·xhigh · GPT-5.4 Pro·xhigh · GPT-5.2·xhigh · GPT-5.2·high · GPT-5.2·medium · GPT-5.2·low · GPT-5.1·low · GPT-5.1·medium · GPT-5.1·high |
| Epoch · frontiermath erdos epoch.ai | Epoch AI's own evaluation (frontiermath_erdos), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Gemini 4 Argon·high · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GPT-6 Sol·max · Claude Opus 5.5·max · GPT-6 Astra·max · Gemini 4 Carbon·high |
| Epoch · frontiermath tier 4 epoch.ai | Epoch AI's own evaluation (frontiermath_tier_4), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.4 mini·high · GPT-5.4 nano·high · GPT-5.4·xhigh · GPT-5.2·xhigh · GPT-5.2·high · GPT-5.2·medium · GPT-5.2·low · GPT-5.1·medium · GPT-5.1·high |
| Epoch · frontiermath tier 4 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | GPT-6.1 Terra·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·max · Claude Sonnet 5.5·max · GPT-6 Luna·max · Grok 4.7·xhigh · GPT-6 Sol·max · Claude Opus 5.5·max · Qwen3.8 Max 0902 build·xhigh · Gemini 3.8 Flash·high · Claude Fable 5.1·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GLM-5.3-Flash·max · GLM-5.3·max · DeepSeek V4 Pro 0813 build·max · Grok 4.6·xhigh · Gemini 3.6 Flash·high · Claude Opus 5·max · Kimi K3·max · Grok 4.20·think · GPT-5.6 Terra·max · GPT-5.6 Luna·max · GPT-5.6 Sol·max · Grok 4.5·high · Claude Sonnet 5·max · Grok 4.3·high · GPT-5.4 Pro·xhigh · GPT-5.2 Pro·xhigh · Kimi K2.7 Code·solo · GPT-5.5 Pro·xhigh · GPT-5.4 nano·high · GPT-5.4 mini·xhigh · GPT-5.4·xhigh · GPT-5.5·xhigh · GPT-5.2·xhigh · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Fable 5·max · Gemini 4 Carbon·high |
| Epoch · frontiermath tiers 1 3 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | GPT-6.1 Terra·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·max · GPT-6 Luna·max · Grok 4.7·xhigh · GPT-6 Sol·max · Claude Opus 5.5·max · Qwen3.8 Max 0902 build·xhigh · Gemini 3.8 Flash·high · Claude Fable 5.1·max · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high · GPT-6 Astra·max · Qwen3.7 Flash·nothink · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GLM-5.2·high · Qwen3.6-27B·nothink · Gemini 3.1 Flash-Lite·minimal · GPT-5.4 nano·nothink · GPT-5.4 mini·nothink · GPT-5.4 nano·low · GPT-5.4 mini·low · GLM-5.3-Flash·max · GLM-5.3·max · DeepSeek V4 Pro 0813 build·max · Grok 4.6·xhigh · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Claude Opus 5·max · Kimi K3·max · Grok 4.20·think · GPT-5.6 Terra·max · GPT-5.6 Luna·max · GPT-5.6 Sol·max · Grok 4.5·high · Claude Sonnet 5·max · Grok 4.3·high · GPT-5.4 Pro·xhigh · GPT-5.2 Pro·xhigh · Kimi K2.7 Code·solo · GPT-5.5 Pro·xhigh · GPT-5.4 nano·high · GPT-5.4 mini·xhigh · GPT-5.4·xhigh · GPT-5.5·xhigh · GPT-5.2·xhigh · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Fable 5·max · Claude Fable 5.2·max · Gemini 4 Carbon·high |
| Epoch · furniture assembly epoch.ai | Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | GPT-6.1 Terra·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·max · GPT-6 Luna·max · Grok 4.5·high · Grok 4.6·xhigh · Grok 4.7·xhigh · GPT-6 Sol·max · Claude Opus 5.5·max · Kimi K3·max · Qwen3.8 Max 0902 build·xhigh · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Claude Fable 5.1·max · Gemini 3.1 Pro·high · Claude Fable 5·max · Claude Opus 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · GPT-5.6 Luna·max · GPT-6 Astra·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-5.5·xhigh · GPT-5.2·xhigh · GPT-5.4·xhigh · Claude Fable 5.2·max · Gemini 4 Carbon·high |
| Epoch · gpqa diamond epoch.ai | Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | GPT-6.1 Terra·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·max · GPT-6 Luna·max · Grok 4.7·xhigh · GPT-6 Sol·max · Claude Opus 5.5·max · Qwen3.8 Max 0902 build·xhigh · Gemini 3.8 Flash·high · GPT-6 Astra·max · Qwen3.5-35B-A3B·nothink · GLM-5.3-Flash·max · GLM-5.3·max · DeepSeek V4 Pro 0813 build·max · Grok 4.6·xhigh · Grok 4.6·high · GLM-5.2·high · GPT-5.4 mini·xhigh · GPT-5.6 Sol·nothink · GPT-5.5·nothink · Kimi K3·low · Kimi K3·high · GPT-5.6 Terra·nothink · GPT-5.4 mini·nothink · GPT-5.6 Sol·low · Kimi K2.7 Code·solo · GPT-5.6 Luna·nothink · GPT-5.6 Terra·low · Qwen3.6-27B·nothink · Gemini 3.6 Flash·minimal · GPT-5.6 Luna·low · Qwen3.7 Flash·nothink · GPT-5.4 nano·nothink · GPT-5.4 nano·low · Gemini 3.6 Flash·low · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Pro·high · Gemini 3.1 Flash-Lite·minimal · Gemini 3.1 Flash-Lite·high · Gemini 3.1 Flash-Lite·low · Gemini 3 Flash·high · Claude Sonnet 5·max · Claude Opus 5·low · Claude Opus 4.8·low · Claude Opus 4.7·max · Claude Fable 5·max · Claude Fable 5·low · Claude Fable 5·high · Gemini 3.6 Flash·high · Claude Opus 5·max · Kimi K3·max · GPT-5.4·medium · GPT-5.4·nothink · GPT-5.4·high · GPT-5.4·low · Claude Sonnet 4.6·high · GPT-5.2·nothink · Grok 4.20·think · GPT-5.6 Terra·max · GPT-5.6 Luna·max · GPT-5.6 Sol·max · Grok 4.5·high · Claude Sonnet 5·xhigh · Grok 4.3·high · Claude Opus 4.8·max · GPT-5.5·low · Claude Opus 4.7·xhigh · GPT-5.4 mini·high · GPT-5.4 nano·high · GPT-5.4 Pro·xhigh · GPT-5.4·xhigh · GPT-5.2·xhigh · GPT-5.2·high · GPT-5.2·medium · GPT-5.2·low · GPT-5.1·medium · GPT-5.1·high · Claude Fable 5.2·max · Gemini 4 Carbon·high |
| Epoch · mirrorcode epoch.ai | Epoch AI's own evaluation (mirrorcode), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Claude Opus 5.5·max · Gemini 3.8 Flash·high · Grok 4.5·high · Grok 4.6·xhigh · GPT-6 Luna·max · Gemini 4 Argon·high · Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high · Claude Opus 4.7·high · Gemini 3.1 Pro·high · GPT-5.5·high · GPT-5.4·high · GPT-5.6 Sol·high · Claude Fable 5·high |
| Epoch · mystery game puzzles epoch.ai | Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | GPT-6.1 Terra·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·max · GPT-6.1 Sol·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · GPT-6 Luna·max · Grok 4.7·xhigh · GPT-6 Sol·max · Gemini 3.8 Flash·high · Claude Fable 5.1·max · GLM-5.3·max · GPT-5.4·nothink · GPT-5.6 Luna·nothink · GPT-6 Astra·max · Kimi K3·low · Kimi K2.6·nothink · GLM-5.3-Flash·max · GPT-5.6 Terra·medium · GPT-5.2·medium · GPT-5.5·low · GPT-5.4·medium · GPT-5.4·low · GPT-5.6 Sol·low · GPT-5.6 Terra·low · GPT-5.1·medium · GPT-5.2·low · GPT-5.6 Luna·medium · GPT-5.4 nano·high · Qwen3.6-27B·nothink · GPT-5.1·low · GLM-5.2·high · Qwen3.5-122B-A10B·nothink · GPT-5.4 mini·medium · GPT-5.4 mini·low · GPT-5.6 Luna·low · Qwen3.7 Flash·nothink · GPT-5.4 nano·medium · GPT-5.6 Sol·nothink · GPT-5.5·nothink · GPT-5.2·nothink · GPT-5.4 nano·low · GPT-5.6 Terra·nothink · GPT-5.4 mini·nothink · GPT-5.4 nano·nothink · DeepSeek V4 Pro 0813 build·max · Grok 4.6·xhigh · GPT-5.2·high · Claude Sonnet 4.6·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·low · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·low · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·low · Gemini 3.6 Flash·high · Claude Fable 5·max · Kimi K3·max · Claude Sonnet 5·max · GPT-5.6 Luna·max · GPT-5.6 Terra·max · GPT-5.6 Sol·max · GPT-5.5·high · Gemini 3.1 Pro·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 5·max · Claude Opus 4.7·max · GPT-5.4·xhigh · GPT-5.5·xhigh · Claude Fable 5.2·max · Gemini 4 Carbon·high |
| Epoch · otis mock aime 2024 2025 epoch.ai | Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·high · GPT-6 Luna·max · Grok 4.7·xhigh · Gemini 3.8 Flash·high · Qwen3.5-35B-A3B·nothink · GLM-5.3-Flash·max · GLM-5.3·max · DeepSeek V4 Pro 0813 build·max · Grok 4.6·xhigh · Grok 4.6·high · GLM-5.2·high · GPT-5.4 mini·xhigh · GPT-5.6 Sol·nothink · GPT-5.5·nothink · Kimi K3·low · Kimi K3·high · GPT-5.6 Terra·nothink · GPT-5.4 mini·nothink · GPT-5.6 Sol·low · GPT-5.5·low · Kimi K2.7 Code·solo · GPT-5.6 Terra·low · GPT-5.6 Luna·nothink · Qwen3.6-27B·nothink · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·low · Qwen3.7 Flash·nothink · GPT-5.6 Luna·low · GPT-5.4 nano·nothink · GPT-5.4 nano·low · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.5 Flash-Lite·high · Gemini 3.1 Flash-Lite·minimal · Gemini 3.1 Flash-Lite·low · Gemini 3.1 Flash-Lite·high · Gemini 3 Flash·high · Claude Sonnet 5·max · Claude Opus 5·low · Claude Opus 4.8·low · Claude Opus 4.7·max · Claude Fable 5·low · Gemini 3.6 Flash·high · Claude Opus 5·max · Kimi K3·max · GPT-5.4·medium · GPT-5.4·nothink · GPT-5.4·high · GPT-5.4·low · Claude Sonnet 4.6·high · GPT-5.2·nothink · Grok 4.20·think · GPT-5.6 Terra·max · GPT-5.6 Luna·max · Grok 4.5·high · Claude Sonnet 5·xhigh · Grok 4.3·high · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.4 mini·high · GPT-5.4 nano·high · GPT-5.4·xhigh · GPT-5.2·xhigh · GPT-5.2·medium · GPT-5.2·high · GPT-5.2·low · GPT-5.1·low · GPT-5.1·medium · GPT-5.1·high · Claude Fable 5.2·max |
| Epoch · simpleqa verified epoch.ai | Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | GPT-6.1 Terra·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·max · GPT-6 Luna·max · Grok 4.7·xhigh · GPT-6 Sol·max · Claude Opus 5.5·max · Qwen3.8 Max 0902 build·xhigh · Gemini 3.8 Flash·high · Claude Fable 5.1·max · GPT-6 Astra·max · GLM-5.3·max · Kimi K3·max · Kimi K2.7 Code·solo · DeepSeek V4 Pro 0813 build·max · Gemini 3.6 Flash·high · Gemini 3 Flash·high · Grok 4.6·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·xhigh · GPT-5.2·low · GPT-5.2·medium · GPT-5.4 nano·high · GPT-5.4 mini·high · GPT-5.2·high · GPT-5.5·xhigh · GPT-5.4 Pro·xhigh · GPT-5.1·high · GPT-5.4·xhigh · Claude Sonnet 5·xhigh · GPT-5.2·xhigh · Claude Sonnet 5·max · Claude Opus 4.7·xhigh · Claude Opus 4.8·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-5.6 Luna·max · Claude Opus 5·max · Claude Fable 5·xhigh · Claude Sonnet 4.6·high · Gemini 3.1 Pro·high · Claude Fable 5.2·max · Gemini 4 Carbon·high |
| Epoch · swe bench verified epoch.ai | Epoch AI's own evaluation (swe_bench_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Claude Opus 4.7·max · GPT-5.4·high · GPT-5.3-Codex·high · GPT-5.1·high · GPT-5.2·high |
| Epoch AI HLE external table (accuracy %, 2026-10-15 zip) epoch.ai | epoch.ai/data/benchmark_data.zip (hle_external.csv; zip read 2026-10-15; copy gc/c4/src/epoch/2026-10-15); Humanity's Last Exam, accuracy with standard error; Notes column empty (provenance unstated); values differ from every Scale HLE capture (not a restatement); read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·xhigh · Claude Sonnet 5.5·xhigh · GPT-5.4·xhigh |
| epoch-ext-algotune epoch.ai | — | xmodel | GPT-5.2·medium · GPT-5.4·high |
| epoch-ext-btf3 epoch.ai | — | xmodel | Claude Opus 4.8·xhigh · Claude Fable 5·high · GPT-5.5·high · GPT-5.6 Sol·high · Claude Opus 4.8·high · Claude Sonnet 5·xhigh · Claude Opus 5·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-5.6 Sol·medium |
| epoch-ext-deepresearchbench epoch.ai | — | xmodel | Claude Sonnet 4.6·high · GPT-5.5·high · Claude Sonnet 4.6·low · Claude Opus 4.8·high · Gemini 3 Flash·low · GPT-5.5·medium · Claude Opus 4.8·low · Gemini 3 Flash·minimal · GPT-5.5·low · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Claude Opus 4.8·medium · Gemini 3.1 Pro·low · GPT-5.1·low · GPT-5.2·low · Gemini 3.1 Flash-Lite·low · GPT-5.4 mini·low · GPT-5.4·low |
| epoch-ext-enigma-eval epoch.ai | — | xmodel | GPT-5.4·xhigh · Claude Fable 5·high · GPT-5.6 Sol·high · Claude Opus 4.8·xhigh |
| epoch-ext-metr-time-horizons epoch.ai | — | xmodel | GPT-5.2·high · GPT-5.4·xhigh |
| epoch-ext-surface-evolver-bench epoch.ai | — | xmodel | Qwen3.8-27B·xhigh · Claude Fable 5·high · GPT-5.6 Sol·xhigh · GPT-5.5·high · Kimi K3·max · Claude Opus 4.8·high · GPT-5.6 Terra·xhigh · GPT-5.5·medium · Grok 4.5·high · Kimi K2.7 Code·solo · GLM-5.2·high · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · GLM-5.3-Flash·max · Gemini 3.8 Flash·high · DeepSeek V4.1 Flash·high |
| epoch-spireclimb-sts1-a20-heart epoch.ai | — | xmodel | Claude Opus 5.5·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6.1 Terra·max · Claude Fable 5.2·max |
| epoch-spireclimb-sts2-a10 epoch.ai | — | xmodel | GPT-6.1 Terra·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · Claude Opus 5.5·max · GPT-6 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·max |
| EQ-Bench creative-writing-longform (EQ-bench/eqbench-leaderboard-results) EQ-bench/eqbench-leaderboard-results (GitHub) | GitHub repo EQ-bench/eqbench-leaderboard-results, pinned commit 266853f2ac7d10c31430cb3bd00d64c758beb581, dir creative-writing-longform/results_sonnet-4.6/, 12/12 multi-chapter longform tasks, judge claude-sonnet-4-6 (flag claude-judge), 1 iteration (n=1); distinct board from the already-held eqbench4 (eqbench.com static roleplay-Elo data, different judge/metric/source) — not a restatement.; read 2026-10-08 ✓ | xmodel | Grok 4.20·think · Kimi K2 Thinking·solo |
| eval-forge-lost-temple-P github-eval-forge | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.2·high · Claude Opus 5.5·high · Claude Opus 5.5·medium |
| eval-forge-lost-temple-S github-eval-forge | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.2·high · Claude Opus 5.5·high · Claude Opus 5.5·medium |
| eval-forge-lost-temple-sxp github-eval-forge | — | xmodel | Claude Opus 5.5·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.2·high · Claude Opus 5.5·high · Claude Opus 5.5·medium |
| EvoClaw evo-claw.com | EvoClaw, long-horizon repository evolution in Claude Code, current board v1.0.2 · overall score in %, total cost of the run (USD) and output tokens; effort from the board's label (Opus 4.8 max 1M, Opus 4.7 xhigh; none stated otherwise); the Opus 4.7 1M-context and OpenHands rows left out (another configuration); read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.6 Sol·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh |
| Expert-SWE (Internal) (OpenAI GPT-5.5 post chart) openai | Expert-SWE (Internal); OpenAI GPT-5.5 launch post chart per reasoning effort (none..xhigh); no cost on this chart (tokens/tool calls only); published 2026-04-23 (snapshot Wayback 20260423180657); read 2026-10-19 ✓ | xmodel | GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh |
| ExploitBench (OpenAI GPT-5.6 post chart) openai | ExploitBench cap percent, API harness, 5 seeds, reasoning continuity (fn1); web chart JSON, not the system-card image; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| ExploitGym 2h cap (OpenAI GPT-5.6 post chart) openai | ExploitGym intended-exploit rate, 2h time cap; latencies rescaled from an alpha API (fn2); OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·xhigh · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| EXTRA osworld2-offline-batch500 osworld-v2-board | new group made by verifier v6 from the g6 report (rows flagged extra-group; no held Claude rows unless stated); read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| EXTRA retroboard-antigravity github-retrospective-board-eval | new group made by verifier v6 from the g6 report (rows flagged extra-group; no held Claude rows unless stated); read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.1 Pro·low |
| EXTRA tb21-codex github-harbor-framework-terminal-bench-2-1 | new group made by verifier v6 from the g6 report (rows flagged extra-group; no held Claude rows unless stated); read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max |
| Factual accuracy on difficult prompts, stored as 100 - error rate (OpenAI GPT-6 Sol and Luna post) openai | Share of answers WITHOUT any factual error on de-identified difficult prompts (stored as 100 - the plotted error rate, lower-is-better in the source); OpenAI GPT-6 Sol and Luna post chart per effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; read 2026-10-19 ✓ | xmodel | GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| Factual accuracy on difficult prompts, stored as 100 - error rate (OpenAI GPT-6.1 Sol post) openai | Share of answers WITHOUT any factual error on de-identified difficult prompts (stored as 100 - the plotted error rate, lower-is-better in the source); OpenAI GPT-6.1 Sol post chart per effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| fcode20pass cognition | — | xmodel | DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Kimi K3·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| fcode20score cognition | — | xmodel | Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · GLM-5.3·low · GLM-5.3·high · GLM-5.3·max · GLM-5.3-Flash·low · GLM-5.3-Flash·high · GLM-5.3-Flash·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high · Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Kimi K3·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| fcodev1ext cognition | — | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · Claude Opus 4.7·xhigh · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·high · GPT-5.4 mini·xhigh |
| FinanceAgent v1.1 (table) openai | FinanceAgent v1.1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Financial Touchstone v1.2 accuracy (annual-report QA, shared top-5 retrieval) arxiv-2608.08634 | arXiv 2608.08634 (9 Aug 2026) Financial Touchstone v1.2: 2,967 question-context-answer items on 495 annual reports, same top-5 retrieved context for every model, output cap 5,000 tokens, LLM judge Claude Opus 4.6 (16 models) / Claude Opus 4.8 (Kimi K2.6, GLM-4.7, Qwen3-Max, ERNIE 5.1); score = accuracy excluding retriever errors (Fig. 1); hallucination rate = same runs, other metric (not a group); read 2026-10-20 ✓ | xmodel | Kimi K2.6·think · GLM-5·think · GLM-4.7·think |
| FinancialAuditBench (2609.32835) — 45 audit workpaper tasks, bash-only agent, 8 runs/task arxiv-2609.32835 | Table 8/Table 9 main row, strict pass % and mean cost $/trajectory, 360 trajectories/model; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·medium · Grok 4.6·medium · GPT-5.6 Sol·medium · GPT-6 Astra·medium · GPT-5.6 Terra·medium · GLM-5.2·high · Gemini 3.8 Flash·medium · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| FinIndices Single-Index with formula hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; with formula hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Single-Index with formula hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; with formula hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Single-Index with formula hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; with formula hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Single-Index without hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; without hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Single-Index without hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; without hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Single-Index without hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; without hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Table-Index with formula hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; with formula hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Table-Index with formula hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; with formula hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Table-Index with formula hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; with formula hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Table-Index without hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; without hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Table-Index without hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; without hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | GLM-5·think |
| FinIndices Table-Index without hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; without hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | GLM-5·think |
| Firecrawl 57-run firecrawl | claude -p in sandbox-exec · 3 models at matched high, measured usage; 7/7 all → cost only ✓ | xmodel | Claude Fable 5·high · Claude Fable 5.1·high · Claude Opus 5·high |
| fireworks-dsflash-deepswe fireworks-blog | — | xmodel | GPT-6 Astra·xhigh · Gemini 3.8 Flash·high · Claude Opus 5·max |
| FlavourBench Score (mean of 3 families x 178 tasks) flavourbench-2608.20574 | FlavourBench Table 3 leaderboard, 534 tasks, FB Score = equal-family mean, 2 collection panels pooled, OpenRouter fixed provider (GLM 5.3 direct); arXiv 2608.20574 (20 Aug 2026); family columns Table 10 are same runs (rule 8); read 2026-10-20 ✓ | xmodel | Grok 4.6·low · Gemini 3.1 Pro·low · Claude Fable 5·low · Gemini 3.6 Flash·minimal · Qwen3.8 Max·low · GLM-5.3·low |
| ForecastBench preliminary leaderboard: dataset-question Brier Index, web search + code execution, 128k, 984 questions forecastbench | ForecastBench preliminary leaderboard; dataset score = difficulty-adjusted Brier Index; N=984 resolved dataset questions; cost absent; read 2026-10-18 ✓ | xmodel | Claude Opus 5·max · Claude Fable 5·max · Claude Fable 5·high · GPT-5.6 Sol·max |
| forecastbench-tourn-ws128k-overall forecastbench | — | xmodel | Claude Opus 5·max · Claude Fable 5·max · GPT-5.6 Sol·max |
| FrontierCode (OpenAI GPT-6 Sol and Luna post) openai | FrontierCode; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| FrontierCode 1.1 Extended (OpenAI GPT-6 Astra post chart, code mode) openai | FrontierCode 1.1 Extended; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = code; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| FrontierCode 1.1 Extended (OpenAI GPT-6 Astra post chart, non-code mode) openai | FrontierCode 1.1 Extended; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| FrontierCode v1.1 Extended (xAI Grok 4.6 post) xai | FrontierCode v1.1 (Extended) %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · GPT-5.6 Sol·max · Claude Fable 5·max |
| FrontierFinance T4 arxiv-2608.11683 | arXiv 2608.11683 Table 4 · effort omitted, Fable 5 vs Opus 4.8 ✓ | xmodel | Claude Fable 5·high · GPT-5.6 Sol·medium · Gemini 3.6 Flash·medium · GPT-5.5·medium · Gemini 3.1 Pro·high |
| FrontierMath Tier 1–3 (table) openai | FrontierMath Tier 1–3; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| FrontierMath Tier 4 (table) openai | FrontierMath Tier 4; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| FrontierMath Tier 4 (v2) (OpenAI GPT-6 Astra post chart, code mode) openai | FrontierMath Tier 4 (v2); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = code; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| FrontierMath Tier 4 (v2) (OpenAI GPT-6 Astra post chart, non-code mode) openai | FrontierMath Tier 4 (v2); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh |
| FrontierSWE dominance score (GLM-5.2 card, as of 2026/06/16) zhipu | FrontierSWE, evaluated by Proximal, 1M context 128K output, dominance score as of 2026/06/16; held group frontierswe is mean-reward@5 (other metric and date); read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · Gemini 3.1 Pro·high |
| FrontierSWE dominance score (GLM-5.3 card, as of 2026/08/14) zhipu | FrontierSWE, evaluated by Proximal, 1M context, 128K output, dominance score as of 2026/08/14; held group frontierswe is mean-reward@5 (glm-5.3 30.18 there vs 78.1 here: different metric); read 2026-10-19 ✓ | xmodel | GLM-5.3·max · Claude Opus 4.8·max · Claude Fable 5·max |
| FrontierSWE v2 frontierswe | Proximal frontierswe.com + Sonnet 5.5 card · 34 ultra-long tasks, max effort, proximus harness, mean reward@5; cost basis undocumented ✓ | xmodel | Claude Fable 5.1·max · GPT-5.6 Sol·max · GLM-5.3·max · Kimi K3·max · Grok 4.6·xhigh · Gemini 3.7 Flash·high · Gemini 4 Carbon·high |
| gameasg-strict-codex-cli arxiv-2609.21293 | — | xmodel | GPT-6 Astra·max · GPT-5.6 Sol·max |
| GameLogicBench arxiv-2609.21562 | arXiv 2609.21562 · 72 Godot game-logic tasks in Claude Code, effort=high, Opus 5 vs Sonnet 5, tokens × vendor list ✓ | xmodel | Claude Opus 5·high · Claude Sonnet 5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·high |
| garrytan/gbrain-evals N2 judge replay (6 Oct 2026): 741 judged pairs, contradiction detection prompt v4; recall saturated 150/150 for all three, cost-only github-gbrain-evals | docs/benchmarks/2026-10-06-n2-judges; cost from the repo budget ledger (741 paid requests); no effort control; read 2026-10-18 ✓ | xmodel | Claude Fable 5.1·high · GPT-6.1 Sol·medium |
| GDP.pdf (OpenAI GPT-6.1 Sol post) openai | GDP.pdf; OpenAI GPT-6.1 Sol post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; Wayback first capture; read 2026-10-19 ✓ | xmodel | GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| GDPval (Elo) (xAI Grok 4.7 post) xai | GDPval Elo (Artificial Analysis GDPval-AA family by Elo scale); chart only; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Claude Fable 5.1·max · Grok 4.7·xhigh · Grok 4.6·high · GPT-6 Astra·max |
| GDPval (wins or ties) (table) openai | GDPval (wins or ties); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| GDPval-AA v2 (OpenAI GPT-5.6 post chart) openai | GDPval-AA v2; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · Gemini 3.5 Flash·high · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Claude Opus 4.8·max |
| GDPVal-AA v2 (xAI Grok 4.6 post) xai | GDPVal-AA v2 Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · GPT-5.6 Sol·max · Claude Fable 5·max |
| gdpx-overall semianalysis | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Gemini 4 Carbon·high · GPT-6 Astra·max · Claude Fable 5.1·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·max · Gemini 4 Argon·high · GPT-6 Luna·max · Grok 4.7·xhigh · GLM-5.3·max · GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-arcverex semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max |
| gdpx-p-aria semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max |
| gdpx-p-arithmetic semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-beautiful-machines semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-chakra semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max |
| gdpx-p-datacurve semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max |
| gdpx-p-emulated semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-ergodex semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-extrapolate semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-fleet semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max |
| gdpx-p-latchbio semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-living-context semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max |
| gdpx-p-third-moment semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| gdpx-p-valenceforge semianalysis | — | xmodel | GPT-6.1 Terra·max · Claude Haiku 5.5·max · Kimi K3·max |
| Gemini 3 Deep Think blog table (12 Feb 2026): cmt googleblog-deepthink-feb26 | blog.google Deep Think post, evaluation table image; cmt; maker-reported, no cost; read 2026-10-20 ✓ | xmodel | Gemini 3 Pro·high · GPT-5.2·xhigh |
| Gemini 3 Deep Think blog table (12 Feb 2026): icho2025 googleblog-deepthink-feb26 | blog.google Deep Think post, evaluation table image; icho2025; maker-reported, no cost; read 2026-10-20 ✓ | xmodel | Gemini 3 Pro·high · GPT-5.2·xhigh |
| Gemini 3 Deep Think blog table (12 Feb 2026): imo2025 googleblog-deepthink-feb26 | blog.google Deep Think post, evaluation table image; imo2025; maker-reported, no cost; read 2026-10-20 ✓ | xmodel | Gemini 3 Pro·high · GPT-5.2·xhigh |
| Gemini 3 Deep Think blog table (12 Feb 2026): ipho2025 googleblog-deepthink-feb26 | blog.google Deep Think post, evaluation table image; ipho2025; maker-reported, no cost; read 2026-10-20 ✓ | xmodel | Gemini 3 Pro·high · GPT-5.2·xhigh |
| GeneBench (OpenAI GPT-5.5 post chart) openai | GeneBench; OpenAI GPT-5.5 launch post chart per reasoning effort (none..xhigh); no cost on this chart (tokens/tool calls only); published 2026-04-23 (snapshot Wayback 20260423180657); read 2026-10-19 ✓ | xmodel | GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh |
| GeneBench Pro (OpenAI GPT-5.6 post chart) openai | GeneBench Pro, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| Gert Labs GBENCH agentic coding (models write code run as agents) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode agentic_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Kimi K2.7 Code·solo · Grok 4.20·think |
| Gert Labs GBENCH decision making (models play the games) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode decision_making, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Kimi K2.7 Code·solo · Grok 4.20·think |
| Gert Labs GBENCH one-shot coding (models write code in one shot) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode oneshot_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Kimi K2.7 Code·solo · Grok 4.20·think |
| github-telecom-oam-objective-off github-telecom-oam | — | xmodel | Qwen3.8 Flash·nothink |
| github-telecom-oam-scenario-off github-telecom-oam | — | xmodel | Qwen3.8 Flash·nothink |
| github-telecom-oam-scenario-on41 github-telecom-oam | — | xmodel | DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high |
| GM-Bench 2.0 agentic lane (hockey-GM 5-season episode, 32 private seeds, Claude Code) github-gmbench | nedcut/gm-bench README (rows published 12-15 Oct 2026); Claude Code harness default effort; mean score over 32-seed private panel, reference pick-trader heuristic = 249.2; rows flagged unpinned by the source; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium · GPT-6 Luna·medium · GPT-6.1 Sol·low · GPT-6 Astra·low |
| GM-Bench 2.1: 5-season hockey GM inside Claude Code, 32-seed private panel, score-v2 (released 2026-10-16) github-gmbench | nedcut/gm-bench results/agentic_v21 claude-2.1.281 panel-32x5; reference pick-trader 249.618 (2.1 sim-v4-league21); NEW GROUP: 2.1 is a new simulator/panel, held gm-bench2-agentic is 2.0 (pick-trader 250.068); read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Astra·low · GPT-6 Luna·medium · GPT-6.1 Sol·low |
| Google Earth AI GRA paper: CBGB, GRA googleearthai-2610.15750 | CBGB test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: CBGB, LLM googleearthai-2610.15750 | CBGB test set, configuration 'LLM', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: CBGB, LLM + Basic Tools googleearthai-2610.15750 | CBGB test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: CBGB, LLM + GRA Tools googleearthai-2610.15750 | CBGB test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: GeoExpertQueries, GRA googleearthai-2610.15750 | GeoExpertQueries test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: GeoExpertQueries, LLM + Basic Tools googleearthai-2610.15750 | GeoExpertQueries test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: GeoExpertQueries, LLM + GRA Tools googleearthai-2610.15750 | GeoExpertQueries test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: GeoKTools, GRA googleearthai-2610.15750 | GeoKTools test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: GeoKTools, LLM googleearthai-2610.15750 | GeoKTools test set, configuration 'LLM', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: GeoKTools, LLM + Basic Tools googleearthai-2610.15750 | GeoKTools test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: GeoKTools, LLM + GRA Tools googleearthai-2610.15750 | GeoKTools test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: UnivEARTH, GRA googleearthai-2610.15750 | UnivEARTH test set, configuration 'GRA', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: UnivEARTH, LLM googleearthai-2610.15750 | UnivEARTH test set, configuration 'LLM', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: UnivEARTH, LLM + Basic Tools googleearthai-2610.15750 | UnivEARTH test set, configuration 'LLM + Basic Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google Earth AI GRA paper: UnivEARTH, LLM + GRA Tools googleearthai-2610.15750 | UnivEARTH test set, configuration 'LLM + GRA Tools', identical queries, 3 runs mean, 100 LLM calls / 30 min limits, failed runs score 0; metric 0-100 score; cost = estimated LLM token cost per query (input, cached, output, thinking tokens at model pricing current at evaluation time, October 2026; external data/compute excluded); arXiv 2610.15750 Tables 4/9 (11 Oct 2026); vendor paper; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium |
| Google model cards: APEX-Agents | APEX-Agents long horizon professional tasks; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3 Pro·high · GPT-5.2·xhigh |
| Google model cards: BrowseComp (search + python + browse) | BrowseComp, Gemini via Deep Research with search, python and browsing; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3 Pro·high · GPT-5.2·xhigh |
| Google model cards: CharXiv Reasoning (1000 val questions) | CharXiv Reasoning, 1000 reasoning questions of the validation split; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| Google model cards: GDPval-AA (Elo) | GDPval-AA Elo (Artificial Analysis); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3 Pro·high · GPT-5.2·xhigh |
| Google model cards: GPQA Diamond (no tools) | GPQA Diamond no tools; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| Google model cards: LiveCodeBench (175 UI problems) | LiveCodeBench code generation, 175 UI problems 1 Jan-1 May 2025; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| Google model cards: MCP Atlas (public set, Turing) | MCP Atlas public set sourced from Turing; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3 Pro·high · GPT-5.2·xhigh |
| Google model cards: SciCode | SciCode (Artificial Analysis); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high · GPT-5.2·xhigh |
| Google model cards: tau2-bench Retail | tau2-bench Retail (Airline excluded); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·xhigh |
| Google model cards: tau2-bench Telecom | tau2-bench Telecom (Airline excluded); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·xhigh |
| Google model cards: Terminal-Bench 2.0 (best self-reported harness) | Terminal-Bench 2.0, other best self-reported harness (Codex); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · GPT-5.3-Codex·xhigh |
| Google model cards: Video-MMMU | Video-MMMU; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Grok 4.1 Fast·think |
| GPQA Diamond (OpenAI GPT-6 Astra post chart) openai | GPQA Diamond; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| GPQA Diamond (table) openai | GPQA Diamond; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| GPQA-Diamond (GLM-4.7 card) zhipu | GPQA-Diamond; GLM-4.7 card 2026-01-29; settings: default settings temp 1.0 top-p 0.95 131072; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| Graphwalks BFS 1mil f1 (table) openai | Graphwalks BFS 1mil f1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Graphwalks BFS 256k f1 (table) openai | Graphwalks BFS 256k f1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Graphwalks parents 1mil f1 (table) openai | Graphwalks parents 1mil f1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Graphwalks parents 256k f1 (table) openai | Graphwalks parents 256k f1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| GRIP closed-loop accuracy (E1, mean over Q1-Q5), spatial and physical reasoning for VLMs grip-2610.16170 | GRIP Table 3 Mean (E1): 34 domains, procedurally generated images with closed-form answers, 5 questions per image in fixed order, fresh single-turn per item, 8,500 queries per hosted model, routed API; hosted max output 256-2048 tokens; Q1-Q5 columns are the same runs (rule 8); arXiv 2610.16170 (Oct 2026); read 2026-10-20 ✓ | xmodel | Gemini 3.8 Flash·low · Claude Opus 5·low · Grok 4.6·low · GPT-5.6 Sol·nothink · Claude Sonnet 5·low · GPT-5.6 Luna·nothink |
| habr-crashtest27 habr-author | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high · Gemini 4 Argon·medium |
| Haiku 5.5 launch page: Terminal-Bench 4.0, cache-read price-cut illustration (7 Oct 2026) anthropic | anthropic.com/claude-haiku-5-5 launch page, 'Further updates' section, embedded chart CSV (exact); read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| Haiku 5.5 system card: biomedical image analysis, max effort anthropic | CT/MRI/histopathology/dermoscopy/retinal/ultrasound/microscopy imaging, scaled to best public-competition result; 1 attempt/problem (exception to the chapter default); read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5·max · Gemini 3.8 Flash·high · GPT-6 Luna·max |
| Haiku 5.5 system card: FrontierSWE v2, Proximal's own harness, max effort anthropic | 34 ultra-long-horizon tasks, 20h trial budget, mean of 5 trials/task, Proximal's own harness; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Fable 5·max · GPT-5.6 Sol·max |
| Haiku 5.5 system card: Humanity's Last Exam (with tools) score+cost by effort, 5 efforts anthropic | chart image, y-axis printed score labels per point; x-axis (cost, log scale) has printed tick labels ($0.01,$0.02,$0.03,$0.05,$0.07,$0.10,$0.20,$0.30,$0.50,$0.70,$1,$2) but no per-point numeric cost label, so cost is digitised from the marker pixel position (calibration in notes.md: x=1685.88+709.69*log10(cost), fit from the $0.01 and $1 ticks, cross-checked exact at $0.1); Haiku 4.5's single point's x could not be confidently isolated from the crowded pink-diamond legend swatch vs. data point in the time available, left without cost; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| Haiku 5.5 system card: LatchBio Bioinformatics, SingleCellBench, max effort anthropic | single-cell RNA sequencing analysis; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5·max |
| Haiku 5.5 system card: LatchBio Bioinformatics, SpatialBench Verified, max effort anthropic | spatial transcriptomics / chromatin-accessibility analysis; bash tool, file editor, prespecified packages; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5·max |
| Haiku 5.5 system card: medicinal chemistry (ADME), max effort, xmodel anthropic | 504 questions, drug-like molecule structures, ADME property comparison; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Gemini 3.8 Flash·high · GPT-6 Luna·max |
| Haiku 5.5 system card: morphology-to-molecule matching (Axiom Bio), max effort anthropic | matches 4 multi-channel cell-imaging sets to 4 shuffled molecule-dose perturbations, human liver in-vitro assay; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5.5·max · Gemini 3.8 Flash·high · GPT-6 Luna·max |
| Haiku 5.5 system card: protein design, Library Ranking, max effort anthropic | prioritize protein designs for experimental testing; 3 attempts/problem (exception to the chapter default); read 2026-10-08 ✓ | xmodel | Claude Sonnet 5·max · Claude Sonnet 5.5·max |
| Haiku 5.5 system card: protein design, Sequence Generation, max effort anthropic | novel protein sequence generation under design constraints; 1 attempt/problem (exception to the chapter's 5-attempt default, stated explicitly); read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max |
| Haiku 5.5 system card: Protocols, Troubleshooting, max effort anthropic | detect and fix issues in molecular biology lab protocols; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · GPT-6 Luna·max |
| Haiku 5.5 system card: Protocols, Understanding (V2, Benchling), max effort anthropic | real lab protocol document + practical application question; read 2026-10-08 ✓ | xmodel | Claude Sonnet 5·max · Claude Sonnet 5.5·max · Gemini 3.8 Flash·high · GPT-6 Luna·max |
| Haiku 5.5 system card: Terminal-Bench 4.0, safeguards-on, each model's own best-effort config anthropic | Claude Code --bare mode; Haiku 5.5/Haiku 4.5 10 trials/task (660), Sonnet/Opus 5.5 5 trials/task (330); safeguard-flagged trials counted as failures; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5.5·xhigh |
| Haiku 5.5 system card: Terminal-Bench-Science 0.1, safeguards-on, max effort anthropic | Claude Code --bare mode, max thinking effort for all except Haiku 4.5 (fixed budget); 700 trials for Haiku 5.5/4.5; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·max · Claude Sonnet 5.5·max · Claude Opus 5.5·max |
| Haiku 5.5 system card: WANDR soft F1 score+cost by effort, 5 efforts anthropic | chart image, y-axis printed score labels per point; x-axis (cost, log scale) ticks $0.005,$0.01,$0.02,$0.05,$0.10,$0.20,$0.50,$1,$2,$5,$10,$20 give calibration x=1230.9+457.94*log10(cost) (fit from $0.005 and $20 ticks, cross-checked exact at $1 and $0.10); several Opus/Sonnet 'Med'/'High' markers were partly occluded by a crossing line of another series and their centroid is a merge of fragment pixels (lower confidence, noted in notes.md); Haiku 4.5's point not confidently isolated, left without cost; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max |
| hama-jp: coastal Viking village (voxel) youtube-hama-jp | Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·high · Claude Sonnet 5.5·high |
| hama-jp: five-story pagoda garden (three.js) youtube-hama-jp | Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·high · Claude Sonnet 5.5·high |
| hama-jp: model railway with interlocking (3h cap) youtube-hama-jp | Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·high · Claude Sonnet 5.5·high |
| Harbor-Index 1.0 (82 tasks), Terminus-2 harness for all models arxiv-2609.04298 | Harbor-Index 1.0: 82 tasks selected from 54 Harbor benchmarks, 3 trials, harness Terminus-2 for every model; pass rate % and approximate per-run cost (USD) of one full Index run; provider default settings (paper: reasoning effort at provider default); models other than GPT-5.5/Opus 4.8/Gemini 3.1 Pro served via OpenRouter; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · Claude Opus 4.8·nothink · Gemini 3.1 Pro·high |
| Harbor-Index 1.0 (82 tasks), vendor-native harness (Codex / Claude Code / Gemini CLI; Claude Code for open models) arxiv-2609.04298 | Harbor-Index 1.0: 82 tasks, 3 trials; harness = vendor-native (Codex for GPT, Claude Code for Claude, Gemini CLI for Gemini, Claude Code for six OpenRouter-served models); pass rate % and approximate per-run cost (USD); read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · Claude Opus 4.8·high |
| Harbor-Index adapter AA-LCR, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AA-LCR evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter Aider Polyglot, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Aider Polyglot evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter AIME 24 & 25, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AIME 24 & 25 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter AlgoTune, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AlgoTune evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter ARC-AGI-2, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ARC-AGI-2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter BFCL, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BFCL evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter BigCodeBench-Hard, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BigCodeBench-Hard evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter BIX-Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BIX-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter CodePDE, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CodePDE evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter CompileBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CompileBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter CRUST-Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CRUST-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter CyberGym, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CyberGym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter DA-Code, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter DA-Code evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter DeepSynth, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter DeepSynth evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter FeatureBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter FeatureBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter FinanceAgent, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter FinanceAgent evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter GAIA, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GAIA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter GAIA2, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GAIA2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter GPQA Diamond, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GPQA Diamond evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter GSO, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GSO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter HumanEvalFix, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter HumanEvalFix evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Gemini 3 Flash·high |
| Harbor-Index adapter Humanity’s Last Exam, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Humanity’s Last Exam evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter IneqMath, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter IneqMath evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter KUMO, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter KUMO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter LAB-Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LAB-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter LawBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LawBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter LiveCodeBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LiveCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter MedAgentBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MedAgentBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter MMAU, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MMAU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter MMMLU, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MMMLU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter Omni-Math, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Omni-Math evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter PIXIU, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter PIXIU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter QCircuitBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter QCircuitBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter QuixBugs, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter QuixBugs evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter Reasoning Gym, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Reasoning Gym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter ReplicationBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ReplicationBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter ResearchCodeBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ResearchCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3 Flash·high |
| Harbor-Index adapter SciCode, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SciCode evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter Seal-0, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Seal-0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SimpleQA, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SimpleQA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SkillsBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SkillsBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SLDBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SLDBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter Spider 2, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Spider 2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SpreadsheetBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SpreadsheetBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter StrongReject, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter StrongReject evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SWE-Bench Pro, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Bench Pro evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SWE-bench Verified, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-bench Verified evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SWE-Bench-Multilingual, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Bench-Multilingual evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SWE-Lancer, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Lancer evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SWE-smith, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-smith evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter SWT Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWT Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter Terminal-Bench 2.0, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Terminal-Bench 2.0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter USACO, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter USACO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| Harbor-Index adapter WideSearch, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter WideSearch evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | GPT-5.4·nothink · Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| harbor-tb21-codex-pass-at-2 github-harbor-framework-terminal-bench-2-1 | — | xmodel | GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max |
| harbor-tb21-codex-pass-at-3 github-harbor-framework-terminal-bench-2-1 | — | xmodel | GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max |
| harbor-tb21-codex-pass-at-4 github-harbor-framework-terminal-bench-2-1 | — | xmodel | GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max |
| harbor-tb21-codex-pass-at-5 github-harbor-framework-terminal-bench-2-1 | — | xmodel | GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max |
| Harvey LAB (Vals) (xAI Grok 4.6 post) xai | Harvey LAB (Vals) %, xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · GPT-5.6 Sol·max |
| Harvey Legal Agent Benchmark (xAI Grok 4.7 post) xai | Harvey Legal Agent Benchmark %, harness/judge not stated; xAI Grok 4.7 post 21 Sep 2026 (4.6 post calls it 'Harvey LAB (Vals)'); read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · GPT-5.6 Sol·max · Claude Fable 5.1·max |
| HealthBench Professional (OpenAI GPT-6 Astra post chart, non-code mode) openai | HealthBench Professional; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5·xhigh · Claude Opus 5·max · Gemini 3.8 Flash·high · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| HealthBench Professional (xAI Grok 4.7 post) xai | HealthBench Professional % (clinical reasoning), grader/variant not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · GPT-5.6 Sol·max · Claude Fable 5.1·max |
| hehee9 Maze Bench Tier 1 (30 mazes) hehee9maze-bcd609c | Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 30 mazes (6 sizes x 5, 4x4..18x18), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status completed); read 2026-10-20 ✓ | xmodel | Claude Fable 5·medium · Claude Fable 5.1·medium · Claude Haiku 5.5·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium · Claude Sonnet 5.5·medium · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·high · GLM-5.3-Flash·high · GPT-5.6 Luna·medium · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Gemini 3.1 Flash-Lite·medium · Gemini 3.1 Pro·medium · Gemini 3.5 Flash·medium · Gemini 3.5 Flash-Lite·medium · Gemini 3.6 Flash·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·medium · Grok 4.5·medium · Kimi K3·max |
| hehee9 Maze Bench Tier 2 (20 mazes) hehee9maze-bcd609c | Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 20 mazes (15x15..24x24, 5 each), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status running); read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·medium · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium |
| Hex DataBench hex.tech | Hex DataBench, 100 analytics tasks in the Hex workspace, rubric-judged · score in %, average cost per task (USD), tokens per task (one total); effort from the row label; Opus 5, Fable 5.1, Fable 5, Opus 4.8, Sonnet 5 at every listed effort (Opus 5.5 in its own group); read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·medium · GPT-6 Sol·xhigh · GPT-6.1 Sol·medium · GPT-6 Sol·max · GPT-6.1 Sol·max · GPT-6 Astra·low · GPT-6 Astra·max · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6 Sol·high · Claude Opus 5.5·low · GPT-6 Astra·medium · GPT-6 Astra·xhigh · GPT-6 Luna·max · GPT-6 Astra·high · GPT-6 Sol·medium · GPT-6 Sol·low · GPT-6.1 Sol·low · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·medium · GPT-5.6 Terra·xhigh · GPT-5.6 Sol·high · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·high · GPT-5.6 Luna·high · GPT-6 Luna·low · GPT-5.6 Sol·low · GPT-5.6 Terra·medium · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · GPT-5.6 Terra·low · Claude Fable 5.1·max · GPT-5.6 Luna·medium · Claude Haiku 5.5·low · Claude Fable 5.1·xhigh · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Opus 5·max · GPT-5.6 Luna·low · Claude Opus 4.8·xhigh · Claude Opus 5·high · Claude Fable 5·max · Claude Opus 5·xhigh · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Fable 5.1·low · GLM-5.3·max · Claude Fable 5·high · Claude Opus 5·low · Claude Opus 4.8·max · Claude Fable 5·low · Claude Fable 5·medium · GLM-5.3·high · Claude Sonnet 5·xhigh · Claude Opus 4.8·high · DeepSeek V4.1 Flash·high · Claude Sonnet 5·high · DeepSeek V4.1 Flash·low · Claude Sonnet 5·medium · GLM-5.2·high · Claude Sonnet 5·max · Claude Opus 4.8·low · Claude Sonnet 5·low · GLM-5.3·low |
| heydonna-rt-v1.1-audio-legal-structure heydonna | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low · Gemini 3.1 Flash-Lite·minimal |
| heydonna-rt-v1.1-text-legal-structure heydonna | — | xmodel | Claude Sonnet 5.5·high · GPT-6.1 Sol·medium |
| hf-kimi-k3.1-pro-aa-briefcase moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-ale-all moonshot | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-ale-cli moonshot | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-apex-agents moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-automationbench-aa moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-babyvision-py moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-benchcad-notools moonshot | — | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-benchcad-tools moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-biomystery moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-browsecomp moonshot | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max |
| hf-kimi-k3.1-pro-business-operation moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-chartography-py moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-charxiv moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-charxiv-py moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-deck-arena moonshot | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-deepswe moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-design-bench moonshot | — | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-exploitbench moonshot | — | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · GLM-5.3·max |
| hf-kimi-k3.1-pro-exploitgym moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max |
| hf-kimi-k3.1-pro-frontier-finance moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-gdppdf moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-gdppdf-py moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-gdpval-aa moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-gpqa-diamond moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-hallu-hard-tools moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-hallu-notools moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-handbook moonshot | — | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-harvey-legal moonshot | — | xmodel | Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-hle-diamond moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-hle-full moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-hle-tools moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-kcb2 moonshot | — | xmodel | GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-kcb3 moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-mmmupro moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-mmmupro-py moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-officeqa-pro moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-officeqa-pro-v2 moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-osworld-v21 moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-perceptionbench moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-posttrainbench moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-programbench moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-swemarathon moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-tb4 moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-tbs moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · GLM-5.3·max |
| hf-kimi-k3.1-pro-xdev-hard moonshot | — | xmodel | GLM-5.3·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-zerobench-avg5 moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-zerobench-pass5 moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hf-kimi-k3.1-pro-zerobench-py-pass5 moonshot | — | xmodel | Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| hlb-deep-swe-claude-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-codex joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-hermes joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-kimi-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-mini-swe-agent joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-openclaw joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-opencode joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-pi joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-deep-swe-qwen-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-claude-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-codex joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-hermes joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-kimi-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-mini-swe-agent joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-openclaw joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-opencode joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-pi joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-featurebench-qwen-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-claude-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-codex joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-hermes joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-kimi-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-mini-swe-agent joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-openclaw joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-opencode joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-pi joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| hlb-terminal-bench-2.1-qwen-code joelniklaus-harness | — | xmodel | GLM-5.3-Flash·high · GLM-5.3-Flash·low |
| HLE (GLM-4.7 card) zhipu | HLE; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| HLE text-only (GLM-5.2 card) zhipu | HLE text-only subset; HLE, temperature=1.0 top_p=0.95, 163,840 gen tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high |
| HLE w/ tools (GLM-4.7 card) zhipu | HLE w/ tools; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| HMMT Feb 2025 (GLM-4.7 card) zhipu | HMMT Feb 2025; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| hotkv-banking77-fresh hotkv | — | xmodel | Claude Fable 5.1·low · Claude Opus 5.5·low |
| HouseholdBench: household-economics prediction, macro-F1 on categorical targets arxiv-2610.07563 | 32 household tasks, pre-cutoff test split, default reasoning effort; Table 3 of arXiv 2610.07563; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5·high · Claude Opus 4.8·nothink · Qwen3.6-27B·nothink |
| Humanity's Last Exam (no tools) (table) openai | Humanity's Last Exam (no tools); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Humanity's Last Exam (with tools) (table) openai | Humanity's Last Exam (with tools); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Humanity's Sixth Sense (HSS), single-pass API, pass@1 arxiv-2610.08966 | 522 image/video tasks, pass@1 = every rubric criterion met, effort = highest honoured per Table 5; tokens_out = mean thinking + answer tokens per task; read 2026-10-15 ✓ | xmodel | GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6 Sol·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-5.6 Luna·max · GPT-6 Luna·max · Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · Claude Fable 5·max · Claude Opus 5·max · Claude Sonnet 5·max · Claude Opus 4.8·max · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high |
| Hy-MultiTurn (Chinese deep multi-turn dialogue understanding): Importance score hymultiturn-2607.29196 | Hy-MultiTurn leaderboard (Tencent Hunyuan authors; Hy3/Doubao rows are not in the eight providers): six dimensions, final-turn response scored against weighted rubric items; Importance (weighted correct share) = primary metric, mean of three independent rolls; arXiv 2607.29196 (31 Jul 2026); read 2026-10-20 ✓ | xmodel | GPT-5.5·high · Grok 4.5·high · Kimi K3·max · GPT-5.4·high · Gemini 3.1 Pro·high · Claude Opus 4.7·high · Kimi K2.6·think · Kimi K2.6·nothink |
| hypfrontier-challenge64-hf-valid hypfrontier-2608.10843 | — | xmodel | GPT-5.6 Sol·xhigh · GPT-5.6 Terra·xhigh · Claude Fable 5·medium · Grok 4.5·high · GPT-5.6 Luna·xhigh · Kimi K2.7 Code·solo |
| iam_lima: weekly-report-from-notes task, mean cost/time of 3 runs note-iam_lima | note.com/iam_lima/n/ncfed4a80e005; Claude Code 2.1.284, work-notes with a planted weekday error and a planted arithmetic error, 3 runs per setting, means reported directly by the source; effort-from-config: "effortはAPIの既定にあたるhighにそろえ、Sonnet 5.5だけはClaude Codeの既定のmediumでも回しました"; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Sonnet 5.5·medium · Claude Sonnet 5·high |
| ibmartian-frontier10-p-claude ironbee | — | xmodel | Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Fable 5.1·xhigh |
| ibmartian-frontier10-p-gpt ironbee | — | xmodel | GPT-6 Astra·low · GPT-6 Astra·xhigh |
| ibmartian-full-p-claude ironbee | — | xmodel | Claude Sonnet 5.5·low · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5·low · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Opus 5.5·xhigh · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Sonnet 5·xhigh · Claude Haiku 5.5·medium · Claude Sonnet 5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·low · Claude Sonnet 5·low |
| ibmartian-full-p-gemini ironbee | — | xmodel | Gemini 4 Argon·high · Gemini 4 Argon·medium · Gemini 4 Argon·low |
| ibmartian-full-p-gpt ironbee | — | xmodel | GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-5.6 Terra·xhigh · GPT-6 Luna·medium · GPT-5.6 Terra·low · GPT-6 Luna·xhigh · GPT-6.1 Sol·xhigh · GPT-6 Luna·low |
| ibmartian-full-p-grok ironbee | — | xmodel | Grok 4.7·low · Grok 4.7·high · Grok 4.7·xhigh |
| ibmartian-full-p-other ironbee | — | xmodel | Kimi K3·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4 Pro 0813 build·high · DeepSeek V4.1 Flash 1017 build·low · Qwen3.8 Max 0902 build·low · DeepSeek V4 Pro 0813 build·low · GLM-5.3·high · Qwen3.8 Max 0902 build·xhigh · GLM-5.3·low |
| ibragim-codex ibragim-dev | — | xmodel | GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6 Sol·medium |
| IdentityBench v1: identity-governance decisions (680 test cases), Claude CLI baseline prompt github-identitybench | rlwillen0121/identity-eval-harness README leaderboard; frozen test split, baseline prompt, provider claude_cli; no effort set; score = grounded accuracy %; cost = API-equivalent USD per 1,000 decisions; read 2026-10-15 ✓ | xmodel | GPT-6.1 Sol·medium · GPT-6 Astra·medium · Gemini 4 Argon·medium · Claude Sonnet 5.5·medium · Claude Fable 5.1·high · Gemini 3.8 Flash·medium · Claude Opus 5.5·medium · DeepSeek V4.1 Flash·high · GPT-6 Luna·medium · Gemini 3.5 Flash-Lite·minimal |
| IdentityBench v3-Full: 360 multi-turn stateful IGA episodes (max 12 turns), claude_cli provider, 17 Oct 2026 github-identitybench | rlwillen0121/identity-eval-harness README fad1304 22-model v3_full table; experiments/v3-full.yaml; NEW GROUP (held identitybench-v1 is the 680-case single-prompt v1 suite); read 2026-10-18 ✓ | xmodel | GPT-6.1 Sol·medium · Gemini 4 Argon·medium · Claude Opus 5.5·medium · GPT-6 Astra·medium · Claude Sonnet 5.5·medium · Claude Fable 5.1·high · GPT-6 Luna·medium · DeepSeek V4.1 Flash·high · Gemini 3.8 Flash·medium · Grok 4.7·high |
| IdentityBench v4.1 v41_lite github-identitybench | IdentityBench v4.1 v41_lite: 60 enterprise IGA scenarios x 6 governance categories = 360 category tasks, 1,628 required actions, 12-turn cap (T0 single monolith), seed 20261019, 1 run per model, README of 2026-10-20 (commit 56591d0), published 2026-10-19; metric tasks solved of 360; cost = run cost of the whole suite (billed or estimated); read 2026-10-20 ✓ | xmodel | Claude Haiku 5.5·medium · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · DeepSeek V4.1 Flash·high · Gemini 3.8 Flash·medium · Gemini 4 Argon·medium · GLM-5.3-Flash·max · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6.1 Sol·medium · Grok 4.7·high |
| identitybench-v41skill-a1 github-identitybench | — | xmodel | Claude Haiku 5.5·medium · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · Gemini 3.8 Flash·medium · Gemini 4 Argon·medium · GPT-6.1 Sol·medium |
| identitybench-v41skill-a2 github-identitybench | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium · Gemini 3.8 Flash·medium · Gemini 4 Argon·medium |
| identitybench-v41skill-a3 github-identitybench | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium · Gemini 3.8 Flash·medium · Gemini 4 Argon·medium |
| identitybench-v41skill-a4 github-identitybench | — | xmodel | Claude Haiku 5.5·medium · Gemini 3.8 Flash·medium · Gemini 4 Argon·medium · GPT-6.1 Sol·medium |
| IH-Benchmark instruction-hierarchy compliance, conflict scenarios (overall %) ih-benchmark-2607.25987 | IH-Benchmark Table 16 Overall: 2,336 conflict scenarios (S>U and U>T tracks), programmatic checks + 189 LLM-judge scenarios (GPT-5-mini judge), once per model, LiteLLM, max_completion_tokens 8192, provider-default sampling, Feb-May 2026; HiddenLayer; arXiv 2607.25987 (27 Jul 2026); Tables 15/17 and track/domain columns = same runs (rule 8); cost = B.3 representative cost per full-benchmark evaluation; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.4·low · GPT-5.2·medium · GPT-5.2·low · GPT-5.4·nothink · Grok 4.20·think · Grok 4.20·nothink |
| IndustryBench Chinese, Final (SV), non-thinking vs thinking arxiv-2605.10267 | IndustryBench (arXiv 2605.10267 v3): 2,049 Chinese industrial-procurement QA items, zero-shot closed-book, raw 0-3 by Qwen3-Max judge minus safety-violation penalty (Final SV, 0-3); outputs collected Feb 2026; Table 7 = default non-reasoning run, Table 8 = same items/judge with thinking enabled; read 2026-10-19 ✓ | xmodel | GPT-5.4·nothink · GPT-5.2·nothink · Claude Sonnet 4.6·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink |
| InferenceBench (prompt v1.0.1 (†)) inferencebench.ai | InferenceBench, agents tuning LLM inference serving in Claude Code · geometric-mean speed-up over the PyTorch eager baseline across four scenarios, ±SEM over seed-pair runs in the flags; runs with the benchmark prompt v1.0.1 (†) only; effort from the model name; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Fable 5·low · Kimi K2.7 Code·solo |
| InferenceBench (prompt v1.0) inferencebench.ai | InferenceBench, agents tuning LLM inference serving in Claude Code · geometric-mean speed-up over the PyTorch eager baseline across four scenarios, ±SEM over seed-pair runs in the flags; runs with the benchmark prompt v1.0 only; effort from the model name; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·xhigh · GPT-5.5·xhigh · GPT-5.5·high · Claude Fable 5·low |
| IntelligentDocumentProcessing: 250 synthetic German documents, invoice routing correct (single-model systems) github-idp-invoices | DanielHuette/IntelligentDocumentProcessing README, measured 8 Oct 2026; one shared prompt; no extended thinking for Claude; cost = tokens billed x list price per 1,000 documents; read 2026-10-15 ✓ | xmodel | GPT-5.6 Terra·low · GPT-5.6 Luna·low · Claude Haiku 5.5·nothink |
| Internal Research Debugging Evaluation (OpenAI GPT-5.6 post chart) openai | Internal Research Debugging Evaluation, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| Investment Banking Modeling Tasks (Internal) (table) openai | Investment Banking Modeling Tasks (Internal); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| ITBench-AA (SRE) artificialanalysis | Artificial Analysis ITBench-AA (SRE); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash·max · Gemini 3.8 Flash·high · GLM-5.1·think · GLM-5.3·max · GLM-5.3-Flash·max · GPT-5.4 mini·xhigh · GPT-5.4 mini·nothink · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Sol·max · Grok 4.3·high · Grok 4.7·xhigh · Kimi K2.6·think · Kimi K3·max · Grok 4.1 Fast·nothink · Claude Opus 4.7·max · Gemini 3.5 Flash·high · GPT-5.4 nano·xhigh · GPT-5.4 nano·nothink |
| izumo-review-high-detection qiita-m_izumo | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| Japanese Stroke LLM Eval arxiv-2609.16739 | multi-turn Japanese conversation · Fable5/Opus4.7 default, score-only ✓ | xmodel | Claude Fable 5·high · Claude Opus 4.7·high |
| JockeyBench Program track (324 races, daily revision agent) arxiv-2610.19589 | Claude Code with MCP bridge; log gain in nats relative to uniform, mean over runs; no model cost (fees are betting score-yen); read 2026-10-15 ✓ | xmodel | GPT-6.1 Sol·high · GPT-6 Astra·high · Claude Opus 5·high · GPT-5.6 Sol·high · GPT-6 Sol·high · Claude Fable 5.1·high · Claude Opus 5.5·high · GPT-5.6 Luna·high · GPT-6 Luna·high · GPT-5.6 Terra·high |
| K2.5 report Table 4: AdvancedIF moonshot | AdvancedIF; K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| K2.5 report Table 4: MMMU (val) moonshot | MMMU (val); K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| K2.5 report Table 4: MMVP moonshot | MMVP; K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| K2.5 report Table 4: OSWorld-Verified moonshot | OSWorld-Verified; K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| K2.5 report Table 4: SimpleQA Verified moonshot | SimpleQA Verified; K2.5 technical report Table 4 (p.12), row absent from the model card; temperature 1.0, top-p 0.95, 256k context; read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh |
| K3 report Fig.13b: BrowseComp score vs cost per task moonshot | BrowseComp; K3 with context compaction at 300K tokens; GPT-5.6 Sol per effort medium/high/max (points read from the vector drawing of Fig.13b); cost USD per task as plotted (source of costs not stated); K3 tech report p.32; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·max · Kimi K3·max · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·max · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·max |
| K3 report Table 3: Agent Behavior Bench moonshot | Agent Behavior Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| K3 report Table 3: Chat All-in-One Bench moonshot | Chat All-in-One Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| K3 report Table 3: Coding Experience [Claude Code harness] moonshot | Coding Experience (Moonshot in-house); harness Claude Code; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max |
| K3 report Table 3: Coding Experience [Codex harness] moonshot | Coding Experience (Moonshot in-house); harness Codex; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · GPT-5.5·xhigh |
| K3 report Table 3: DECK Bench moonshot | DECK Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| K3 report Table 3: Faithfulness moonshot | Faithfulness (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| K3 report Table 3: Finance Bench moonshot | Finance Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| K3 report Table 3: KWV Bench moonshot | KWV Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| K3 report Table 3: MIRA Bench moonshot | MIRA Bench (Moonshot in-house); harness MIRA; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| kamiumi-svg-motion note-kamiumi_ai | — | xmodel | Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| kenji_harada: 4 hard coding tasks, Opus 5.5 sweep + Fable 5.1 (Claude Code) qiita-kenji-harada | Claude Code 2.1.283 -p, macOS, 4 hard Node.js tasks x3 runs, median time/cost at API list price; qiita.com/kenji_harada/items/7d8be259208cd3a5843f; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Fable 5.1·high |
| kenji_harada: 6 small coding tasks, Opus 5.5/Opus 5/Fable 5.1 (Claude Code) qiita-kenji-harada | Claude Code 2.1.283 -p, macOS, 6 small Node.js tasks x3 runs, median time/cost at API list price; qiita.com/kenji_harada/items/7d8be259208cd3a5843f; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·low · Claude Opus 5·high · Claude Fable 5.1·high |
| KernelBench CUDA 01 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 01, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Claude Haiku 5.5·max · Claude Opus 4.8·max · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Luna·xhigh · GPT-6 Sol·xhigh · GPT-6.1 Sol·xhigh · Grok 4.7·xhigh · Grok 4.6·xhigh · Claude Fable 5.2·high |
| KernelBench CUDA 02 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 02, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Claude Haiku 5.5·max · Claude Opus 4.8·max · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Luna·xhigh · GPT-6 Sol·xhigh · GPT-6.1 Sol·xhigh · Grok 4.7·xhigh |
| KernelBench CUDA 03 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 03, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Claude Haiku 5.5·max · Claude Opus 4.8·max · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Luna·xhigh · GPT-6 Sol·xhigh · GPT-6.1 Sol·xhigh · Grok 4.7·xhigh · Grok 4.6·xhigh · Claude Fable 5.2·high |
| KernelBench CUDA 03 (OpenRouter) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 03, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Claude Fable 5·high · Claude Opus 5·max |
| KernelBench CUDA 04 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 04, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Claude Haiku 5.5·max · Claude Opus 4.8·max · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Luna·xhigh · GPT-6 Sol·xhigh · GPT-6.1 Sol·xhigh · Grok 4.7·xhigh |
| KernelBench CUDA 04 (OpenRouter) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 04, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Claude Fable 5·high · Claude Opus 5·max |
| KernelBench hard 01_fp8_gemm github-kernelbench | kernelbench.com hard board problem 01_fp8_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh · Claude Fable 5·high · Claude Opus 4.8·high · GPT-5.6 Sol·xhigh · Claude Fable 5·max |
| KernelBench hard 03_paged_attention github-kernelbench | kernelbench.com hard board problem 03_paged_attention; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh · Claude Fable 5·high · Claude Opus 4.8·high |
| KernelBench hard 05_topk_bitonic github-kernelbench | kernelbench.com hard board problem 05_topk_bitonic; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh · Claude Fable 5·high · Claude Opus 4.8·high · GPT-5.6 Sol·xhigh · Claude Opus 5·max |
| KernelBench hard 07_w4a16_gemm github-kernelbench | kernelbench.com hard board problem 07_w4a16_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Grok 4.6·xhigh · Claude Fable 5·high · Claude Opus 4.8·high · GPT-5.6 Sol·xhigh · Claude Opus 5·max · Claude Opus 4.8·max |
| KernelBench MEGA (Claude Code) github-kernelbench | 02_kimi_linear_decode megakernel on RTX PRO 6000 Blackwell · harness=claude (Claude Code) · n=1 per cell, runs months apart (Jun-Sep 2026); NOT the board's CUDA task (that is the held kernelbench-cc/or group); read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · Claude Fable 5·high · Claude Fable 5.1·high · Claude Opus 4.8·high · GPT-6.1 Sol·low · GPT-5.5·medium · GPT-5.6 Sol·low · Kimi K2.7 Code·solo · GPT-6 Luna·medium · GPT-6 Sol·medium |
| KernelGen 1P (OpenAI GPT-5.6 post chart) openai | KernelGen 1P, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| Kimi Code Bench 2.0 moonshot | arXiv 2607.24653 Fig. 13a · Moonshot in-house suite via Claude Code, Opus 4.8 vs Fable 5 at max, measured cost (vector-read) ✓ | xmodel | Claude Opus 4.8·max · Claude Fable 5·max |
| Kimi K2.5 card: AIME 2025 moonshot | AIME 2025; avg@32; 96k tokens; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: BrowseComp moonshot | BrowseComp; no context management (60.6 without vs 74.9 with Discard-all, K2.5 report p.11); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: BrowseComp (w/ctx manage) moonshot | BrowseComp (w/ctx manage); discard-all context management (K2.5 and DeepSeek-V3.2); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: CharXiv (RQ) moonshot | CharXiv (RQ); vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: CyberGym moonshot | CyberGym; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: DeepSearchQA moonshot | DeepSearchQA; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: FinSearchCompT2&T3 moonshot | FinSearchCompT2&T3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: GPQA-Diamond moonshot | GPQA-Diamond; avg@8; 96k tokens; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: HLE-Full moonshot | HLE-Full; full set (text+image); 96k completion budget; HF blocked; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: HMMT 2025 (Feb) moonshot | HMMT 2025 (Feb); avg@32; 96k tokens; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: IMO-AnswerBench moonshot | IMO-AnswerBench; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: InfoVQA (val) moonshot | InfoVQA (val); vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: LiveCodeBench (v6) moonshot | LiveCodeBench (v6); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: Longbench v2 moonshot | Longbench v2; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: LongVideoBench moonshot | LongVideoBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: LVBench moonshot | LVBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: MathVision moonshot | MathVision; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: MathVista (mini) moonshot | MathVista (mini); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: MMLU-Pro moonshot | MMLU-Pro; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: MMMU-Pro moonshot | MMMU-Pro; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: MotionBench moonshot | MotionBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: OCRBench moonshot | OCRBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: OJBench (cpp) moonshot | OJBench (cpp); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: PaperBench moonshot | PaperBench; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh |
| Kimi K2.5 card: SciCode moonshot | SciCode; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: Seal-0 moonshot | Seal-0; avg@4; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: SimpleVQA moonshot | SimpleVQA; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: SWE-Bench Pro moonshot | SWE-Bench Pro; in-house framework (bash/createfile/insert/view/strreplace/submit); temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh |
| Kimi K2.5 card: VideoMME moonshot | VideoMME; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: WideSearch (item-f1) moonshot | WideSearch (item-f1); avg@4; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | Gemini 3 Pro·high |
| Kimi K2.5 card: ZeroBench moonshot | ZeroBench; vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.5 card: ZeroBench (w/ tools) moonshot | ZeroBench (w/ tools); vision max tokens 64k; avg@3; temperature 1.0, top-p 0.95, 256k context; Kimi K2.5 model card (HF created 2026-01-01, announcement 2026-01-27); read 2026-10-19 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Pro·high |
| Kimi K2.6 card: AIME 2026 moonshot | AIME 2026; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: APEX-Agents moonshot | APEX-Agents; 452 of the 480 public tasks (as Artificial Analysis); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: BabyVision moonshot | BabyVision; vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: BabyVision (w/ python) moonshot | BabyVision (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: BrowseComp (Agent Swarm) moonshot | BrowseComp (Agent Swarm); discard-all context management; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: CharXiv (RQ) moonshot | CharXiv (RQ); vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: CharXiv (RQ) (w/ python) moonshot | CharXiv (RQ) (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: Claw Eval (pass^3) moonshot | Claw Eval (pass^3); Claw Eval v1.1; max tokens per step 16,384; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: DeepSearchQA (f1-score) moonshot | DeepSearchQA (f1-score); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: HLE-Full moonshot | HLE-Full; full set; max generation 98,304 tokens; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: HLE-Full (w/ tools) moonshot | HLE-Full (w/ tools); full set; max generation 98,304 tokens; tools: search/code/browse; per-step 49,152; context management; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: HMMT 2026 (Feb) moonshot | HMMT 2026 (Feb); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: IMO-AnswerBench moonshot | IMO-AnswerBench; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: LiveCodeBench (v6) moonshot | LiveCodeBench (v6); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · Gemini 3.1 Pro·high |
| Kimi K2.6 card: MathVision moonshot | MathVision; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: MathVision (w/ python) moonshot | MathVision (w/ python); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: MCPMark moonshot | MCPMark; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: MMMU-Pro moonshot | MMMU-Pro; vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: MMMU-Pro (w/ python) moonshot | MMMU-Pro (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: OJBench (python) moonshot | OJBench (python); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · Gemini 3.1 Pro·high |
| Kimi K2.6 card: OSWorld-Verified moonshot | OSWorld-Verified; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh |
| Kimi K2.6 card: SciCode moonshot | SciCode; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: SWE-Bench Multilingual moonshot | SWE-Bench Multilingual; in-house SWE-agent-derived framework; avg of 10 runs; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · Gemini 3.1 Pro·high |
| Kimi K2.6 card: Toolathlon moonshot | Toolathlon; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.6 card: V* (w/ python) moonshot | V* (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think · GPT-5.4·xhigh · Gemini 3.1 Pro·high |
| Kimi K2.7 Code card: Kimi Claw 24/7 Bench moonshot | Kimi Claw 24/7 Bench; in-house; 17 scenarios/610 evaluation points; OpenClaw harness; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo · GPT-5.5·xhigh · Claude Opus 4.8·xhigh |
| Kimi K2.7 Code card: Kimi Code Bench v2 moonshot | Kimi Code Bench v2; in-house; 10+ languages; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo · GPT-5.5·xhigh · Claude Opus 4.8·xhigh |
| Kimi K2.7 Code card: MCP Atlas moonshot | MCP Atlas; 100 tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo · GPT-5.5·xhigh · Claude Opus 4.8·xhigh |
| Kimi K2.7 Code card: MCP Mark Verified moonshot | MCP Mark Verified; human-verified MCPMark; 100-step tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo · GPT-5.5·xhigh · Claude Opus 4.8·xhigh |
| Kimi K2.7 Code card: MLS Bench Lite moonshot | MLS Bench Lite; 30-task subset; 5 h per task; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo · GPT-5.5·xhigh · Claude Opus 4.8·max |
| Kimi K2.7 Code card: Program Bench moonshot | Program Bench; 200 tasks; 248k fuzz tests; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo · GPT-5.5·xhigh · Claude Opus 4.8·xhigh |
| Kimi K3 card: AA-Briefcase (Elo) moonshot | AA-Briefcase Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: AA-LCR moonshot | AA-LCR; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: Agents' Last Exam moonshot | Agents' Last Exam; official leaderboard as of 2026-07-23, primary pass-rate metric; per-model harness from the leaderboard; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·xhigh · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: APEX-Agents moonshot | APEX-Agents; Mercor leaderboard as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: AutomationBench moonshot | AutomationBench; 600-task public subset; official GitHub setup; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: BabyVision w/ python moonshot | BabyVision with Python; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: CharXiv (RQ) (no tools) moonshot | CharXiv reasoning questions; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: CharXiv (RQ) (with python) moonshot | CharXiv reasoning questions; avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: FrontierSWE moonshot | FrontierSWE; dominance scores recomputed from raw scores, current as of 2026-07-16; K3 Kimi Code, Sol Codex; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: GDPval-AA v2 (Elo) moonshot | GDPval-AA v2 Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: GPQA Diamond moonshot | GPQA Diamond; no tools; top-p 0.95; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: HLE-Full (no tools) moonshot | HLE full set; top-p 0.95; with-tools = general tools; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: HLE-Full (with tools) moonshot | HLE full set; top-p 0.95; with-tools = general tools; with tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MathVision (no tools) moonshot | MathVision; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MathVision (with python) moonshot | MathVision; avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MCP-Atlas moonshot | MCP-Atlas; 500-task public subset; 100-turn limit; judge Gemini 3.1 Pro; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MCPMark-Verified moonshot | MCPMark-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MLS-Bench-Lite moonshot | MLS-Bench-Lite (30-task subset); Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MMMU-Pro (no tools) moonshot | MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MMMU-Pro (with python) moonshot | MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: MMVU moonshot | MMVU; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: OfficeQA Pro moonshot | OfficeQA Pro; entire PDF corpus given as images, no machine-readable text; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: OmniDocBench moonshot | OmniDocBench; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: OSWorld 2.0 moonshot | OSWorld 2.0; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: OSWorld-Verified moonshot | OSWorld-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: PerceptionBench moonshot | PerceptionBench (Moonshot in-house benchmark); avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: PostTrainBench moonshot | PostTrainBench; K3/Fable/Sol official Harbor implementation at max effort, avg of 3 runs on H20 (official: H100); GLM/GPT-5.5/Opus from the official board; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: ProgramBench moonshot | ProgramBench; K3 with Kimi Code harness; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: ResearchRubrics moonshot | ResearchRubrics; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: SaaS-Bench moonshot | SaaS-Bench; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: SciCode moonshot | SciCode; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: SpreadsheetBench 2 moonshot | SpreadsheetBench 2; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: SWE-Marathon moonshot | SWE-Marathon, H20-calibrated branch (card: official tasks as of 2026-07-09 prior to final v1.1; blog/report: v1.1 tasks); K3/Opus/Fable Claude Code, Sol Codex; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: Terminal-Bench 2.1 moonshot | Terminal-Bench 2.1; K3 with Kimi Code harness; others = best score across harnesses; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · GPT-5.5·xhigh |
| Kimi K3 card: Toolathlon-Verified moonshot | Toolathlon-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: Video-MME (w. sub) moonshot | Video-MME with subtitles; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: WorldVQA ForceAnswer moonshot | WorldVQA ForceAnswer variant; vision avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: ZeroBench (pass@5) (no tools) moonshot | ZeroBench main set; pass@5 (run 5 times, official setting); no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: ZeroBench (pass@5) (with python) moonshot | ZeroBench main set; pass@5 (run 5 times, official setting); with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| Kimi K3 card: τ³-Banking moonshot | tau3-Banking; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh |
| kingy930869-4tasks kingy.ai | — | xmodel | GPT-6 Luna·low · Claude Haiku 5.5·low · GPT-6.1 Sol·low · Claude Sonnet 5.5·low · Claude Opus 5.5·low · GPT-6 Astra·low · Claude Fable 5.1·low · Claude Fable 5.2·low · GPT-6 Sol·low · Grok 4.6·low · Grok 4.7·low |
| krauq.ai SCAN 2026 CTF qualifier (55 flags, 1 attempt each) krauq.ai | Claude Code (builds 2.1.273/2.1.284/2.1.293), 55 blockchain-investigation CTF flags, one attempt per flag, 20-min limit; score = flags solved of 55; cost = API-equivalent total of the 55 attempts (sum of per-attempt cost_usd; estimated attempts flagged); read 2026-10-15 ✓ | xmodel | GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Luna·nothink · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·nothink · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5.1·low · Claude Fable 5.1·medium |
| Labelbox · Implicit Intelligence labelbox.com | Labelbox leaderboards, Implicit Intelligence · score as printed (integer), top five models only; effort from the label; read 2026-09-30 ✓ | xmodel | GPT-5.2 Pro·high · Claude Opus 4.7·high · GPT-5.4 Pro·xhigh · Gemini 3 Pro·high · GPT-5.2·high · GPT-5.4·xhigh |
| latitude easy tier github-latitude | ai-sdk · 12 single-file bug tasks ×3, 3 Claude models at default effort, list price ✓ | xmodel | GPT-5.5·medium · Claude Opus 4.8·nothink · GPT-5.6 Sol·medium · GPT-5.6 Luna·medium · GPT-5.6 Terra·medium |
| latitude-cross6 github-latitude | — | xmodel | GPT-5.5·medium · Claude Opus 4.8·nothink · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium |
| latitude-sweep github-latitude | — | xmodel | Kimi K3·max · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh |
| lb2-browser-analysis-driven-action livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-browser-constraint-adherence livebench | — | xmodel | Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Luna·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-browser-data-aggregation livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-browser-sequential-planning livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-browser-structural-integrity livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-coding-agent-loop livebench | — | xmodel | Claude Haiku 5.5·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · GLM-5.3-Flash·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · Grok 4.6·xhigh |
| lb2-coding-billing-limits livebench | — | xmodel | Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Grok 4.6·xhigh · Kimi K3·max |
| lb2-coding-cloud-infrastructure livebench | — | xmodel | Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Sonnet 5.5·max · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · Kimi K3·max |
| lb2-coding-hard-fixes livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-coding-llm-api-integration livebench | — | xmodel | Claude Fable 5.2·xhigh · DeepSeek V4.1 Flash 1017 build·max · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-coding-memory-tools livebench | — | xmodel | Gemini 3.8 Flash·high · Gemini 4 Argon·high · GLM-5.3-Flash·max · GPT-6 Luna·max · Kimi K3·max |
| lb2-coding-product-ux livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-dataanalysis-consecutive-events livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-dataanalysis-table-join livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-dataanalysis-table-reformat livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · Grok 4.6·xhigh · Kimi K3·max |
| lb2-design-documents-docx livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-design-pdf livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-design-presentations-ppt livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-longhorizon-delegated-operations livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-longhorizon-finance-compliance-apps livebench | — | xmodel | Claude Fable 5.1·max · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-longhorizon-long-run-operations livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-longhorizon-operations-commerce-apps livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-longhorizon-performance-optimisation livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-longhorizon-systems-engineering livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-reasoning-logic-with-navigation livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-reasoning-spatial livebench | — | xmodel | Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-reasoning-theory-of-mind livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-reasoning-zebra-puzzles livebench | — | xmodel | Claude Haiku 5.5·max · Gemini 3.8 Flash·high · GLM-5.3-Flash·max · GPT-6 Luna·max · Grok 4.6·xhigh · Grok 4.7·xhigh |
| lb2-research-legal livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-research-regulatory-compliance livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-research-science-evidence livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-research-strategy livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| lb2-research-tax-finance livebench | — | xmodel | Claude Fable 5.1·max · Claude Fable 5.2·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Gemini 4 Carbon·high · Gemini 4 Hydrogen·xhigh · GLM-5.3-Flash·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max |
| leanzero-forge2-medium leanzero | — | xmodel | Claude Opus 5.5·medium · GPT-6.1 Sol·medium · Gemini 4 Argon·medium · Claude Sonnet 5.5·medium |
| lechmazur Adversarial LLM Debate Benchmark — Bradley-Terry rating lechmazur-debate | lechmazur/debate, side-swapped adversarial debates, multi-model judge panel (incl. a Claude judge), Bradley-Terry rating with match counts; read 2026-10-05 ✓ | xmodel | Claude Fable 5.1·high · Claude Fable 5·high · Claude Opus 5·high · Claude Opus 5.5·high · Gemini 4 Argon·high · Claude Opus 4.7·high · GPT-5.6 Sol·high · GLM-5.3·high · Claude Opus 4.8·high · Grok 4.7·high · GPT-6 Astra·high · Grok 4.6·high · Claude Sonnet 5·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro 0813 build·high · Claude Sonnet 4.6·high · GPT-5.4·high · GPT-5.5·high · Gemini 3.8 Flash·high · GPT-5.4·nothink · Grok 4.5·high · Gemini 3.7 Flash·high · Grok 4.20·think · Grok 4.20·nothink |
| lechmazur Creative Writing Benchmark — comparison score lechmazur-writing | lechmazur/writing, matched creative-brief stories judged pairwise by rotating LLM evaluators, Thurstone/BT-style comparison score centered at 0, bootstrap 95% CI; read 2026-10-05 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5·xhigh · Claude Opus 5.5·high · GPT-6 Astra·high · Claude Opus 5·high · GLM-5.3·max · GPT-5.6 Sol·xhigh · GPT-5.5·xhigh · GPT-5.6 Sol·high · GPT-5.4·medium · GPT-5.4·xhigh · Gemini 4 Argon·high · Claude Opus 4.8·xhigh · Grok 4.7·high · GPT-5.2·medium · Gemini 3.8 Flash·high · DeepSeek V4.1 Flash·high · Gemini 3.7 Flash·high · Grok 4.6·high · Grok 4.5·high |
| Lechmazur Elimination Game (TrueSkill mu) lechmazur-elimination-game | multi-player elimination tournament (public/private chat, votes, jury), TrueSkill mu aggregated over random passes; games per model printed; README leaderboard of 61 rows; changelog 'Jan 6, 2026: GPT-5.2, Opus 4.5, Gemini 3 Pro, Gemini 3 Flash, Kimi K2 Thinking, Qwen 3 Max Thinking, Grok 4.1 Fast ... added' (stale board); no cost; read 2026-10-19 ✓ | xmodel | GPT-5.2·medium · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| lechmazur NYT Connections Extended Benchmark — full 940-puzzle board lechmazur-nyt | lechmazur/nyt-connections, 940 puzzles, quadratic-v1 scoring (g/4)^2 per puzzle, objective (not model-judged); read 2026-10-05 ✓ | xmodel | Claude Opus 5·xhigh · Claude Fable 5·high · Claude Opus 5·high · Claude Opus 4.8·xhigh · Claude Fable 5.1·high · Claude Opus 5.5·high · Claude Opus 4.8·high · Claude Sonnet 4.6·high · Claude Sonnet 5·high · Claude Opus 4.7·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·xhigh · GPT-6 Astra·high · Gemini 3.8 Flash·high · GPT-5.5·xhigh · GPT-6.1 Sol·high · GPT-5.5·high · GPT-5.6 Sol·xhigh · GPT-5.5·medium · GPT-5.4·xhigh · GPT-5.6 Sol·high · GPT-5.4·high · GPT-6 Sol·high · DeepSeek V4.1 Flash·high · GPT-5.4·medium · Grok 4.20·think · GPT-5.2·xhigh · Grok 4.6·xhigh · Grok 4.5·high · Grok 4.6·high · GPT-5.6 Terra·high · Grok 4.7·high · GLM-5.2·high · GLM-5.3·high · GPT-5.6 Luna·high · GPT-6 Luna·high · GPT-5.2·high · GPT-5.4 mini·xhigh · GPT-5.2·medium · Gemini 3.5 Flash-Lite·high · GPT-5.2·low · GPT-5.5·nothink · GPT-5.4·nothink · GPT-5.2·nothink · Grok 4.1 Fast·nothink · Grok 4.20·nothink · DeepSeek V4.1 Flash 1017 build·high |
| lechmazur NYT Connections Extended Benchmark — harder 100-puzzle subset lechmazur-nyt | lechmazur/nyt-connections, harder 100-puzzle subset, same quadratic-v1 scoring, objective; read 2026-10-05 ✓ | xmodel | Claude Opus 5·xhigh · Claude Fable 5·high · Claude Opus 4.8·high · Claude Opus 5·high · Claude Opus 4.8·xhigh · Claude Opus 5.5·high · Claude Fable 5.1·high · Claude Sonnet 4.6·high · Claude Sonnet 5·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · GPT-6 Astra·xhigh · Gemini 4 Argon·high · GPT-6 Astra·high · Gemini 3.8 Flash·high · GPT-5.5·xhigh · GPT-6.1 Sol·high · GPT-5.5·high · GPT-5.6 Sol·xhigh · GPT-5.4·xhigh · GPT-5.6 Sol·high · GPT-6 Sol·high · GPT-5.4·high · GPT-5.5·medium · DeepSeek V4.1 Flash·high · GPT-5.4·medium · GPT-5.2·xhigh · Grok 4.20·think · Grok 4.5·high · Grok 4.6·xhigh · Grok 4.6·high · GLM-5.2·high · Grok 4.7·high · GPT-5.6 Terra·high · GLM-5.3·high · GPT-5.2·high · GPT-5.6 Luna·high · GPT-6 Luna·high · GPT-5.4 mini·xhigh · Gemini 3.5 Flash-Lite·high · GPT-5.2·medium · GPT-5.2·low · GPT-5.4·nothink · GPT-5.5·nothink · GPT-5.2·nothink · Grok 4.20·nothink · Grok 4.1 Fast·nothink · DeepSeek V4.1 Flash 1017 build·high · Claude Opus 4.7·high |
| legalforecast-mtd-micro-brier johnhughes3/LegalForecastBench | — | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · GPT-6 Sol·high · Grok 4.6·high · GPT-5.6 Sol·high · GPT-5.6 Luna·high · GPT-6.1 Sol·high · GPT-6 Astra·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·high · Gemini 3.1 Pro·high |
| LEGO-Anything: LEGO-Bench Bird's-Eye reconstruction stress split arxiv-2609.36380 | LEGO-Anything (arXiv 2609.36380, 28 Sep 2026): single-image-to-3D-scene reconstruction by coding agents in Codex; LEGO-Bench Bird's-Eye reconstruction stress split; score = validity-gated final score S (%; V, R, A are components of the same runs); reasoning effort not stated for the main tables (harness default); no cost; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6 Luna·medium |
| LEGO-Anything: LEGO-Bench Indoor split arxiv-2609.36380 | LEGO-Anything (arXiv 2609.36380, 28 Sep 2026): single-image-to-3D-scene reconstruction by coding agents in Codex; LEGO-Bench Indoor split; score = validity-gated final score S (%; V, R, A are components of the same runs); reasoning effort not stated for the main tables (harness default); no cost; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium |
| LEGO-Anything: LEGO-Bench Outdoor split arxiv-2609.36380 | LEGO-Anything (arXiv 2609.36380, 28 Sep 2026): single-image-to-3D-scene reconstruction by coding agents in Codex; LEGO-Bench Outdoor split; score = validity-gated final score S (%; V, R, A are components of the same runs); reasoning effort not stated for the main tables (harness default); no cost; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium |
| LexKairos overall arxiv-2608.09106 | LexKairos: Chinese legal temporal reasoning, 9 sub-tasks; temperature 0, max output 8,000 tokens; Vanilla = direct answer without extra reasoning steps; Thinking Mode = built-in reasoning enabled, thinking effort high; weighted overall; read 2026-10-19 ✓ | xmodel | GPT-5.4·nothink · GPT-5.4·high · Gemini 3 Flash·high |
| LIBERO-Agent (30-task embodied-manipulation suite), general-purpose coding agents arxiv-2609.39507 | LIBERO-Agent (arXiv 2609.39507, 30 Sep 2026): 30-task suite, 3 rollouts per task, full observation profile, no demonstrations, 1800 s per rollout; each model in its own agent harness (Table 5); score = Overall Score (Table 1, 0-100); judgement accuracy, short/long-horizon SR and SC are other columns of the same runs; avg time and tokens reported, no dollar cost; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·high · GPT-5.6 Sol·high · Claude Opus 5·high · Claude Fable 5.1·high |
| LibraryDesignBench (2609.36730) — 15 library-design problems, 3 implementer models solve them arxiv-2609.36730 | Table 1 designers: score (correctness x simplicity composite), $/problem; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | GPT-6 Astra·high · GPT-5.6 Sol·high |
| LifeSciBench (OpenAI GPT-5.6 post chart) openai | LifeSciBench, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| LiveBench 2.0 · Browser Use suite livebench | LiveBench 2.0 Browser Use suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6.1 Terra·max · GLM-5.3-Flash·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high |
| LiveBench 2.0 · Coding suite livebench | LiveBench 2.0 Coding suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · GLM-5.3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6.1 Terra·max · GLM-5.3-Flash·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high |
| LiveBench 2.0 · Data Analysis suite livebench | LiveBench 2.0 Data Analysis suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · GLM-5.3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6.1 Terra·max · GLM-5.3-Flash·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high |
| LiveBench 2.0 · Design suite livebench | LiveBench 2.0 Design suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · GLM-5.3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6.1 Terra·max · GLM-5.3-Flash·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high |
| LiveBench 2.0 · Long-Horizon suite livebench | LiveBench 2.0 Long-Horizon suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · GLM-5.3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6.1 Terra·max · GLM-5.3-Flash·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high |
| LiveBench 2.0 · Reasoning suite livebench | LiveBench 2.0 Reasoning suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · GLM-5.3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6.1 Terra·max · GLM-5.3-Flash·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high |
| LiveBench 2.0 · Research suite livebench | LiveBench 2.0 Research suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Sonnet 5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · GLM-5.3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6.1 Terra·max · GLM-5.3-Flash·max · GPT-6 Luna·max · GPT-5.6 Terra·max · GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Fable 5.2·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·xhigh · Grok 4.7·xhigh · Kimi K3·max · Gemini 4 Hydrogen·xhigh · Gemini 4 Carbon·high |
| livebench-task-amps_hard livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-code_completion livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-code_generation livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-connections livebench | — | xmodel | Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·minimal · GPT-5.1·high · GPT-5.1·low · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-consecutive_events livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-integrals_with_game livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-javascript livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-logic_with_navigation livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-math_comp livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-olympiad livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-paraphrase livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-plot_unscrambling livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-python livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-simplify livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-spatial livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·low · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-story_generation livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-summarize livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-tablejoin livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-tablereformat livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-theory_of_mind livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-typescript livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-typos livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·high · Claude Opus 4.7·xhigh · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-zebra_puzzle livebench | — | xmodel | Gemini 3 Pro·high · Claude Sonnet 4.6·high · Claude Sonnet 4.6·low · Claude Opus 4.7·low · Claude Opus 4.7·medium · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · GPT-5.1·high · GPT-5.1·low · GPT-5.1·medium · GPT-5.1-Codex-Max·high · GPT-5.2·high · GPT-5.2·low · GPT-5.2·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4 mini·medium · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.4 nano·low · GPT-5.4 nano·medium · GPT-5.4 nano·xhigh · GPT-5.5·high · GPT-5.5·medium · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·nothink · Grok 4.20·think · Kimi K2 Thinking·solo · Kimi K2.6·think |
| LiveCodeBench-v6 (GLM-4.7 card) zhipu | LiveCodeBench-v6; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| LLM A-share trading simulation (mean return) github-llm-quant-trading | CollinsLiu615/llm-quant-trading (GitHub, commits 30 Sep 2026): simulated A-share market (GARCH, T+1), 15 stocks, 180 trading days, 100,000 CNY, 2 seeds (Claude Sonnet 5: 1 seed); score = mean total return %; individual hobby benchmark; read 2026-10-20 ✓ | xmodel | DeepSeek V4.1 Flash·low · DeepSeek V4 Flash 0731 build·high |
| LLM judge accuracy on the alignment set (439 DPO pairs), original prompts arxiv-2610.18122 | pairwise verdict accuracy vs dataset label, CALM prompt, reasoning off; cost = US$ per 1,000 requests over the full 15,195-request design; read 2026-10-15 ✓ | xmodel | GPT-6.1 Sol·low · Gemini 3.8 Flash·low |
| LLM judge accuracy on the fact-related set, original prompts arxiv-2610.18122 | pairwise verdict accuracy vs correct answer, CALM prompt, reasoning off; read 2026-10-15 ✓ | xmodel | GPT-6.1 Sol·low · Gemini 3.8 Flash·low |
| llm_chess — aggregated Elo vs Random + Dragon engine (all Claude models) llm_chess | maxim-saplin/llm_chess, LLM plays Black vs RANDOM_PLAYER and Dragon engine, Elo anchored to Dragon ladder, 95% CI, average_game_cost as cost per game; read 2026-10-05 ✓ | xmodel | Claude Opus 5·high · Claude Sonnet 5·high · Claude Opus 5.5·high · Claude Sonnet 4.6·high · Claude Opus 4.8·high · Claude Haiku 4.5·nothink · Claude Opus 4.7·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · Gemini 3.8 Flash·high · GPT-5.2 Chat (latest)·solo · Grok 4.6·high · GPT-5.2·medium · Gemini 3.1 Pro·high · GPT-5.6 Sol·high · GPT-6.1 Sol·high · Grok 4.3·high · GPT-5.6 Terra·xhigh · Grok 4.20·nothink · GPT-6 Luna·xhigh · GPT-5.4·high · GPT-5.3-Codex·medium · GPT-5.2·low · GPT-5.4·low · Grok 4.1 Fast·nothink · GPT-5.4 nano·high · GPT-6 Astra·xhigh · GPT-5.6 Terra·high · GPT-5.6 Luna·medium · Gemini 3.8 Flash·medium · GPT-6 Sol·high · GPT-5.5·high · GPT-5.6 Luna·high · GPT-5.1·medium · GPT-5.5·medium · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.1·low · Gemini 3.6 Flash·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·xhigh · GPT-6.1 Sol·xhigh · GPT-5.1 Chat (latest)·solo · GPT-5.4 mini·high · GPT-5.4 mini·low · GPT-5.4·medium · Gemini 3.6 Flash·high · GPT-6 Astra·high · GPT-6 Luna·high · Gemini 3.6 Flash·low · GPT-6 Sol·xhigh · Gemini 4 Argon·high · GLM-5.3-Flash·high |
| llm_chess — single 10-game match vs RANDOM_PLAYER (Black win rate) llm_chess | maxim-saplin/llm_chess raw per-match _aggregate_results.json, n=10 games per model, win% computed by us from black_wins/total_games; read 2026-10-05 ✓ | xmodel | Claude Sonnet 5.5·high · Gemini 3.6 Flash·low · GPT-5.4·high · GPT-5.4·low · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.5·high · Grok 4.20·think · Grok 4.3·high · Grok 4.6·high |
| llm-regression product suite 2026-10-03 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-03, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llm-regression product suite 2026-10-04 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-04, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llm-regression product suite 2026-10-05 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-05, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llm-regression product suite 2026-10-06 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-06, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llm-regression product suite 2026-10-07 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-07, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llm-stats.com arena: agents category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), agents category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo · Grok 4.20·think |
| llm-stats.com arena: general category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), general category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo · Grok 4.20·think · Grok 4.20·nothink |
| llm-stats.com arena: reasoning category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), reasoning category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo · Grok 4.20·think |
| llm2014-logic-2026-10-median llm2014-logic | — | xmodel | GPT-6 Astra·xhigh · Gemini 4 Argon·high · GPT-6.1 Sol·xhigh · Claude Sonnet 5.5·xhigh · DeepSeek V4.1 Flash 1017 build·max · Kimi K3·max · DeepSeek V4.1 Flash·max · Gemini 3.8 Flash·high · GLM-5.3·max · DeepSeek V4 Pro 0813 build·max · GLM-5.3-Flash·max · Claude Haiku 5.5·xhigh · Qwen3.8-27B·xhigh · GPT-6 Luna·xhigh · Gemini 3.5 Flash-Lite·high |
| LLMConfBench arxiv-2609.20666 | arXiv 2609.20666 · conformer energy ranking, 27 molecules × 3 prompts via OpenRouter, Opus 5 and Sonnet 5 at high; cost summed from the repo's per-call logs ✓ | xmodel | GPT-6 Astra·high · Claude Opus 5·high · Gemini 3.6 Flash·high · GPT-5.6 Sol·high · Kimi K3·high · Claude Sonnet 5·high · DeepSeek V4 Flash 0731 build·high · Gemini 3 Flash·high |
| llmreg-product-pass-2026-10-03 nikoamoretti-llm-regression | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llmreg-product-pass-2026-10-04 nikoamoretti-llm-regression | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llmreg-product-pass-2026-10-05 nikoamoretti-llm-regression | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llmreg-product-pass-2026-10-06 nikoamoretti-llm-regression | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| llmreg-product-pass-2026-10-07 nikoamoretti-llm-regression | — | xmodel | Claude Opus 5.5·high · Grok 4.7·high |
| LMArena Agent lmarena.ai | Agent Mode real sessions (Code/Chat/Work) · cost=median $/task (14d window), score=Net Improvement composite signal, effort read from model-variant label; live board, read 27 Sep 2026 ✓ | xmodel | GPT-5.5·xhigh · DeepSeek V4.1 Flash·max · GPT-5.4·high · GPT-5.6 Luna·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Opus 5.5·high · Gemini 4 Carbon·high · Claude Sonnet 5.5·max · Claude Sonnet 5.5·high · Claude Fable 5.1·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6.1 Sol·medium · GPT-6.1 Terra·max · GPT-6.1 Terra·medium · Gemini 4 Argon·high · GPT-6 Sol·max · Claude Fable 5·high · Claude Opus 5·max · GPT-5.6 Sol·xhigh · Grok 4.7·xhigh · Gemini 3.8 Flash·high · Claude Sonnet 5·high · DeepSeek V4.1 Flash 1017 build·max · GPT-6 Luna·max · GLM-5.3·max · Kimi K3·max · Claude Haiku 5.5·high · GPT-5.6 Terra·xhigh · GPT-6 Luna·medium · GPT-6 Luna·low · Gemini 4 Hydrogen·high · Grok 4.6·xhigh · Gemini 3.7 Flash·high |
| LMArena Document lmarena.ai | Document arena (Overall, the page's default board: no style control) · blind pairwise votes on answers about uploaded documents, Elo rating, no cost; vote cutoff 13 Sep 2026, so no 5.5 model yet; effort read from the arena's model key (-thinking, -high, -max); live board, read 30 Sep 2026 ✓ | xmodel | Claude Opus 4.8·nothink · Grok 4.5·high · Grok 4.20·think · Claude Fable 5.1·max · Claude Opus 5·high · Claude Fable 5·high · Claude Opus 4.7·high · GPT-5.5·high · Gemini 3.8 Flash·high · GPT-5.6 Sol·xhigh · Gemini 3.7 Flash·high · Claude Opus 4.8·high · GPT-5.6 Terra·xhigh · GPT-6 Astra·max · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Claude Sonnet 5·high · Gemini 3.6 Flash·high · GPT-5.6 Luna·xhigh · Grok 4.6·high · GPT-5.2·high |
| LMArena Search lmarena.ai | Search arena (Overall, the page's default board: no style control) · blind pairwise votes on answers with web search, Elo rating, no cost; each model runs with the arena's search tool (key suffix -search); vote cutoff 24 Aug 2026, no 5.5 model; effort not in the key (default), Fable 5 at high per its display name; live board, read 30 Sep 2026 ✓ | xmodel | Grok 4.5·high · Claude Opus 4.8·nothink · GPT-5.6 Sol·xhigh · Claude Opus 5·high · Claude Fable 5.1·high · GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Fable 5·high · Claude Opus 5.5·medium · GPT-6 Sol·max · GPT-6 Luna·max |
| LMArena Text lmarena.ai | Text arena (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-opus-5.5-high-vertex); live board, read 30 Sep 2026 ✓ | xmodel | Claude Opus 4.8·nothink · Grok 4.20·think · Grok 4.5·high · GPT-6.1 Sol·max · GLM-5.3·max · DeepSeek V4.1 Flash·max · GPT-6 Astra·max · GPT-6 Luna·max · Claude Opus 5.5·high · Claude Sonnet 5.5·xhigh · GPT-5.6 Terra·xhigh · Gemini 4 Argon·high · Claude Fable 5.1·max · Claude Opus 5·max · Claude Opus 5·high · Gemini 4 Hydrogen·high · Gemini 3.8 Flash·high · Claude Fable 5·high · Claude Opus 4.7·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.5 Flash·medium · Kimi K3·max · GPT-5.5·high · GPT-5.4·high · Claude Opus 4.8·high · GPT-5.6 Sol·xhigh · GPT-5.1·high · Gemini 3 Flash·minimal · Claude Sonnet 5·high · GPT-5.6 Luna·xhigh · Grok 4.6·high · GPT-5.2·high · Kimi K2 Thinking·solo · GPT-5.4 mini·high · Grok 4.7·xhigh · Claude Haiku 5.5·high · GPT-6 Sol·max · GPT-5.4 nano·high · Gemini 4 Carbon·high |
| LMArena Text (xAI Grok 4.1 post) xai | LMArena Text Arena Elo as printed by xAI (chart label 'Overall Style Control Elo', text 'overall'; style control unclear), live board snapshot of 17 Nov 2025; no cost; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| LMArena Vision lmarena.ai | Vision arena (Overall, the page's default board: style control) · blind pairwise votes on image-input prompts, Elo rating, no cost; vote cutoff 28 Sep 2026, no 5.5 model yet; effort read from the arena's model key (-thinking, -thinking-16k/32k, -high, -max); live board, read 30 Sep 2026 ✓ | xmodel | Claude Opus 4.8·nothink · Grok 4.5·high · Grok 4.20·think · Claude Fable 5·high · Claude Opus 5.5·high · Claude Fable 5.1·max · Claude Opus 5·high · Gemini 3.8 Flash·high · Claude Opus 4.7·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · GPT-5.4·high · Gemini 3.6 Flash·high · Claude Opus 4.8·high · GPT-5.5·high · GPT-6.1 Sol·max · Claude Sonnet 5.5·xhigh · GPT-5.6 Sol·xhigh · DeepSeek V4.1 Flash·max · GPT-6 Astra·max · Claude Sonnet 5·high · GPT-5.6 Terra·xhigh · Gemini 3 Flash·minimal · Grok 4.6·high · GPT-5.6 Luna·xhigh · GPT-5.1·high · GPT-5.2·high · GPT-5.4 mini·high · GPT-6 Sol·max · Grok 4.7·xhigh · GPT-6 Luna·max · GPT-5.4 nano·high |
| LMArena WebDev lmarena.ai | Code arena, WebDev board (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-sonnet-5-5-high); live board, read 30 Sep 2026 ✓ | xmodel | Grok 4.5·high · Claude Opus 4.8·nothink · Grok 4.1 Fast·think · GPT-6.1 Terra·max · GPT-6.1 Sol·max · Claude Fable 5.2·xhigh · GPT-6 Luna·max · DeepSeek V4.1 Flash 1017 build·max · GLM-5.3·max · Claude Opus 5.5·max · Claude Sonnet 5.5·xhigh · GPT-6 Astra·max · Gemini 4 Carbon·high · Claude Fable 5.1·max · Claude Sonnet 5.5·high · Gemini 4 Hydrogen·high · Claude Opus 5·max · Gemini 4 Argon·high · GPT-6 Sol·max · Kimi K3·max · Claude Opus 5·high · Grok 4.7·xhigh · Claude Fable 5·high · DeepSeek V4.1 Flash·max · Grok 4.6·high · GPT-5.6 Sol·xhigh · Claude Haiku 5.5·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Claude Opus 4.7·high · Claude Opus 4.8·high · Claude Sonnet 5·high · Gemini 3.6 Flash·high · GPT-5.6 Terra·xhigh · GPT-5.6 Luna·xhigh · GPT-5.5·xhigh · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · GPT-5.5·high · Kimi K2.7 Code·solo · GPT-5.4·high · GPT-5.4·medium · GPT-5.4 mini·high · GPT-5.1·medium · Gemini 3 Flash·minimal · Kimi K2 Thinking·solo |
| LMArena WebDev — HTML sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-html, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Opus 5.5·max · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·high · Claude Fable 5.1·max · GPT-6 Sol·max · Claude Opus 5·max · Claude Fable 5·high · Claude Opus 5·high · Kimi K3·max · Gemini 4 Argon·high · Grok 4.7·xhigh · Gemini 3.8 Flash·high · DeepSeek V4.1 Flash·max · Grok 4.6·high · GLM-5.3·max · GPT-6 Luna·max · Gemini 3.7 Flash·high · Grok 4.5·high · Claude Haiku 5.5·high · Gemini 3.6 Flash·high · Claude Opus 4.7·high · Claude Opus 4.8·high · Claude Sonnet 5·high · Claude Opus 4.8·nothink · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Kimi K2.7 Code·solo · GPT-5.4 mini·high · GPT-5.1·medium · Gemini 3 Flash·minimal · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| LMArena WebDev — React sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-react, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·xhigh · GPT-6 Astra·max · Claude Fable 5.1·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·high · Claude Opus 5·max · Gemini 4 Argon·high · GPT-6 Sol·max · Claude Opus 5·high · Kimi K3·max · Grok 4.7·xhigh · GLM-5.3·max · Claude Fable 5·high · Grok 4.6·high · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · GPT-6 Luna·max · Claude Opus 4.7·high · Claude Opus 4.8·high · Grok 4.5·high · Claude Sonnet 5·high · Claude Opus 4.8·nothink · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash·medium · Kimi K2.7 Code·solo · GPT-5.4 mini·high · Gemini 3 Flash·minimal · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| lmcouncil-humanity-s-last-exam lmcouncil | — | xmodel | Gemini 3.1 Pro·high · GPT-5.4·xhigh |
| lmcouncil-vpct-visual-physics-comprehension-test lmcouncil | — | xmodel | GPT-5.2·xhigh · GPT-5.2·high |
| lob-code-fix-merge-intervals github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-code-lru-cache github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-code-parse-duration github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-hard-coin-faces github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-hard-count-multiples github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-hard-lex-topo-order github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-hard-locking-tradeoff github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-hard-parse-csv-line github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-hard-postmortem-rewrite github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-hard-puzzle-hunt github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-hard-weighted-intervals github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh |
| lob-reason-race-order github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-reason-snail-pole github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-reason-trailing-zeros github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-write-container-summary github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-write-https-explainer github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| lob-write-launch-email github-low-or-bust | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·xhigh |
| M-GATE grammar error detection, MCC mean over 30 languages (temperature 0) mgate-2608.03803 | M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.1; 53 models / 82 configurations March-July 2026; linguist-crafted adversarial grammar items, 30 languages, temperature 0, one run; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓ | xmodel | GPT-5.2·high · GPT-5.2·medium · GPT-5.4·nothink · GPT-5.4·high · GPT-5.4·medium · GPT-5.4 mini·high · GPT-5.4 mini·medium · GPT-5.4 nano·high · GPT-5.4 nano·medium · GPT-5.5·high · GPT-5.5·medium · Claude Opus 4.8·nothink · Claude Sonnet 5·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·minimal · DeepSeek V4 Pro preview·high · Grok 4.20·think · Grok 4.3·nothink · GLM-5.2·high |
| M-GATE round-trip translation, hard subset, normalised score mean over 29 targets mgate-2608.03803 | M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.2; 53 models / 82 configurations March-July 2026; 100 English sources round-tripped into 29 languages, hard subset of 50 sentences, judges GPT-5.4 Mini + Claude Haiku 4.5 + Gemini 3.1 Flash Lite at temp 0; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓ | xmodel | GPT-5.2·high · GPT-5.2·medium · GPT-5.4·nothink · GPT-5.4·high · GPT-5.4·medium · GPT-5.4 mini·high · GPT-5.4 mini·medium · GPT-5.4 nano·high · GPT-5.4 nano·medium · GPT-5.5·high · GPT-5.5·medium · Claude Opus 4.8·nothink · Claude Sonnet 5·high · Gemini 3 Flash·high · Gemini 3 Flash·minimal · Gemini 3.1 Pro·high · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·minimal · DeepSeek V4 Pro preview·high · Grok 4.20·think · Grok 4.3·nothink · GLM-5.2·high |
| MA-Bench mission analysis, benchmark condition, weighted score (GPT-5.6 judge), Opus 5 / Fable 5.1 / Sonnet 5 arxiv-2610.19138 | arXiv 2610.19138 Table 4; production judge GPT-5.6; 3000-word cap; 216 briefs (n=30/50/30 per model); read 2026-10-18 ✓ | xmodel | Claude Opus 5·medium · Claude Fable 5.1·medium · Claude Sonnet 5·medium · Grok 4.6·high · GPT-6 Astra·medium · Gemini 3.8 Flash·medium · GPT-5.4·medium · Gemini 3.1 Pro·medium |
| MathArena · aime 2025 matharena.ai | MathArena, aime--aime_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | GPT-5.2·high · GPT-5.2·xhigh · Gemini 3 Flash·high · GLM-5·think · GPT-5.1·high |
| MathArena · aime 2026 matharena.ai | MathArena, aime--aime_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · GPT-5.4·xhigh · GPT-5.2·high · Gemini 3.1 Pro·high · Gemini 3 Flash·high · GLM-5·think · Gemini 3.6 Flash·high · Kimi K3·max · Kimi K2.6·think · Gemini 3.5 Flash·high · GLM-5.2·max |
| MathArena · apex 2025 matharena.ai | MathArena, apex--apex_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · GPT-5.4 Pro·xhigh · Kimi K3·max · Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Kimi K2.6·think · GLM-5.2·max · Gemini 3 Flash·high · GPT-5.2·high · GLM-5·think · GPT-5.1·high |
| MathArena · april matharena.ai | MathArena, arxiv--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-6 Astra·max · GPT-5.5·xhigh · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Kimi K3·max · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · GLM-5.2·max · DeepSeek V4.1 Flash·max |
| MathArena · april matharena.ai | MathArena, arxiv_false--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-6 Astra·max · GPT-5.5·xhigh · Kimi K3·max · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.7 Flash·high · GLM-5.2·max · Gemini 3.6 Flash·high |
| MathArena · august matharena.ai | MathArena, arxiv--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5.5·high · Claude Fable 5.1·low · GPT-6 Astra·max · GPT-6 Astra·low · Gemini 3.8 Flash·high · Kimi K3·max · GPT-6.1 Sol·max · GPT-6 Sol·max · Gemini 4 Argon·high · DeepSeek V4.1 Flash 1017 build·max · Grok 4.7·xhigh · DeepSeek V4.1 Flash·max |
| MathArena · august matharena.ai | MathArena, arxiv_false--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Opus 5.5·high · GPT-6 Astra·max · GPT-6 Astra·low · Kimi K3·max · Gemini 3.8 Flash·high · GPT-6.1 Sol·max · GPT-6 Sol·max · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · Gemini 4 Argon·high · Grok 4.7·xhigh |
| MathArena · brumo 2025 matharena.ai | MathArena, brumo--brumo_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Gemini 3 Flash·high · GLM-5·think · GPT-5.2·high · GPT-5.1·high |
| MathArena · february matharena.ai | MathArena, arxiv--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.4 Pro·xhigh · GPT-5.5·xhigh · Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3.5 Flash·high · GLM-5.2·max · Kimi K2.6·think · GLM-5·think · Gemini 3.1 Pro·low · GPT-5.2·high |
| MathArena · february matharena.ai | MathArena, arxiv_false--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · GPT-5.4·xhigh · GPT-5.2·xhigh · Gemini 3.1 Pro·high · GLM-5·think · Kimi K2.6·think · GLM-5.2·max · Gemini 3.5 Flash·high |
| MathArena · hmmt feb 2025 matharena.ai | MathArena, hmmt--hmmt_feb_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | GPT-5.2·xhigh · GPT-5.2·high · Gemini 3 Flash·high · GLM-5·think · GPT-5.1·high |
| MathArena · hmmt feb 2026 matharena.ai | MathArena, hmmt--hmmt_feb_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · GPT-5.4·xhigh · GPT-5.2·high · Kimi K3·max · Gemini 3.5 Flash·high · Kimi K2.6·think · Gemini 3.1 Pro·high · GLM-5.2·max · Gemini 3 Flash·high · Gemini 3.6 Flash·high · GLM-5·think |
| MathArena · january matharena.ai | MathArena, arxiv--january: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.7·xhigh · GPT-5.4·xhigh · GPT-5.5·xhigh · Kimi K2.6·think · Gemini 3.1 Pro·high · GPT-5.2·high · Gemini 3.1 Pro·medium · Gemini 3 Flash·high · GLM-5·think · Gemini 3.1 Pro·low · GPT-5.2·low |
| MathArena · june matharena.ai | MathArena, arxiv--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 5.5·high · Claude Opus 5·max · Claude Opus 4.8·max · Claude Fable 5.1·low · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Sol·max · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-5.5·xhigh · Kimi K3·max · Grok 4.7·xhigh · Gemini 3.8 Flash·high · Grok 4.5·high · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · DeepSeek V4.1 Flash·max · Gemini 3.6 Flash·high · Gemini 3.5 Flash·high · GLM-5.2·max |
| MathArena · june matharena.ai | MathArena, arxiv_false--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 5·max · Claude Fable 5.1·low · Claude Fable 5·max · Claude Opus 4.8·max · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Astra·low · GPT-6 Sol·max · GPT-5.5·xhigh · GPT-5.6 Sol·max · Grok 4.7·xhigh · Kimi K3·max · DeepSeek V4.1 Flash·max · Grok 4.5·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · GLM-5.2·max · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high |
| MathArena · june matharena.ai | MathArena, arxivlean--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 5·max · Claude Opus 4.8·max · Claude Fable 5·max · GPT-6 Astra·max · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · GLM-5.2·max |
| MathArena · kangaroo 2025 1-2 matharena.ai | MathArena, kangaroo--kangaroo_2025_1-2: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · GPT-5.4·xhigh · Kimi K3·max · Gemini 3.5 Flash·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · GPT-5.2·high · GPT-5.1·high |
| MathArena · kangaroo 2025 11-12 matharena.ai | MathArena, kangaroo--kangaroo_2025_11-12: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Kimi K3·max · GPT-5.4·xhigh · GPT-5.5·xhigh · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high · GPT-5.2·high · Gemini 3.5 Flash·high · GPT-5.1·high |
| MathArena · kangaroo 2025 3-4 matharena.ai | MathArena, kangaroo--kangaroo_2025_3-4: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · GPT-5.4·xhigh · Kimi K3·max · Gemini 3.1 Pro·high · GPT-5.2·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3 Flash·high · GPT-5.1·high |
| MathArena · kangaroo 2025 5-6 matharena.ai | MathArena, kangaroo--kangaroo_2025_5-6: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Gemini 3.5 Flash·high · GPT-5.5·xhigh · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · GPT-5.4·xhigh · Kimi K3·max · GPT-5.2·high · Gemini 3 Flash·high · GPT-5.1·high |
| MathArena · kangaroo 2025 7-8 matharena.ai | MathArena, kangaroo--kangaroo_2025_7-8: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Kimi K3·max · GPT-5.4·xhigh · GPT-5.5·xhigh · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 3 Flash·high · GPT-5.2·high · GPT-5.1·high |
| MathArena · kangaroo 2025 9-10 matharena.ai | MathArena, kangaroo--kangaroo_2025_9-10: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.2·high · Gemini 3.5 Flash·high · GPT-5.5·xhigh · GPT-5.4·xhigh · Gemini 3 Flash·high · GPT-5.1·high |
| MathArena · march matharena.ai | MathArena, arxiv--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · Gemini 3.1 Pro·high · GPT-5.4·xhigh · GLM-5.2·max · Kimi K2.6·think · Gemini 3.5 Flash·high · GLM-5·think |
| MathArena · march matharena.ai | MathArena, arxiv_false--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · GPT-5.4·xhigh · Gemini 3.5 Flash·high · Kimi K2.6·think · Gemini 3.1 Pro·high · GLM-5·think · GLM-5.2·max |
| MathArena · may matharena.ai | MathArena, arxiv--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Fable 5·max · Claude Fable 5.1·max · Claude Opus 5.5·high · Claude Opus 4.8·max · Claude Fable 5.1·low · GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-6 Sol·max · GPT-6 Astra·low · GPT-5.5·xhigh · Kimi K3·max · Grok 4.7·xhigh · DeepSeek V4.1 Flash·max · Gemini 3.8 Flash·high · GLM-5.2·max · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high |
| MathArena · may matharena.ai | MathArena, arxiv_false--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·low · Claude Fable 5·max · Claude Opus 4.8·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6 Astra·low · GPT-6 Sol·max · GPT-5.5·xhigh · Kimi K3·max · Grok 4.7·xhigh · DeepSeek V4.1 Flash·max · Gemini 3.8 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · GLM-5.2·max · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high |
| MathArena · shortlist 2025 matharena.ai | MathArena, apex--shortlist_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · Gemini 3.1 Pro·high · Kimi K3·max · Gemini 3.5 Flash·high · GPT-5.4·xhigh · GPT-5.2·high · Kimi K2.6·think · Gemini 3.6 Flash·high · Gemini 3 Flash·high · GLM-5·think · GLM-5.2·max · GPT-5.1·high |
| MathArena · smt 2025 matharena.ai | MathArena, smt--smt_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | GPT-5.2·xhigh · Gemini 3 Flash·high · GPT-5.2·high · GLM-5·think · GPT-5.1·high |
| matharena-arxiv-false-october matharena.ai | — | xmodel | GPT-6.1 Sol·max · Claude Opus 5.5·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Argon·high |
| matharena-arxiv-false-september matharena.ai | — | xmodel | GPT-6.1 Sol·max · Claude Opus 5.5·high · DeepSeek V4.1 Flash 1017 build·max · DeepSeek V4.1 Flash·max · Gemini 4 Argon·high · Grok 4.7·xhigh · Gemini 3.8 Flash·high |
| matharena-arxiv-october matharena.ai | — | xmodel | GPT-6.1 Sol·max · Gemini 4 Argon·high · Claude Opus 5.5·high · DeepSeek V4.1 Flash 1017 build·max |
| matharena-arxiv-september matharena.ai | — | xmodel | GPT-6.1 Sol·max · Gemini 4 Argon·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · DeepSeek V4.1 Flash 1017 build·max · Grok 4.7·xhigh · DeepSeek V4.1 Flash·max |
| matharena-arxivlean-september matharena.ai | — | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·max · GPT-6.1 Sol·max · DeepSeek V4.1 Flash·max |
| matharena-leanlean-oct26 matharena.ai | — | xmodel | GPT-6.1 Sol·max · Claude Opus 5.5·high · Gemini 4 Argon·high · DeepSeek V4.1 Flash 1017 build·max |
| MazeBench ASCII no tools mazebench | MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, no Python tools; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓ | xmodel | GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 5·max · GPT-6 Sol·max · GPT-5.6 Sol·max · GPT-5.5·xhigh · Claude Opus 4.8·max · GPT-5.4·xhigh · Claude Sonnet 5·max · Gemini 3.1 Pro·high · Grok 4.20·think · Gemini 4 Argon·high |
| MazeBench ASCII with Python tools mazebench | MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, Python tools enabled; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓ | xmodel | GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-5.6 Sol·max · Claude Opus 5·max · Claude Fable 5·max · GPT-6 Sol·max · Claude Fable 5.1·max · GPT-5.6 Terra·max · Gemini 3.8 Flash·high · Claude Haiku 5.5·max · Grok 4.6·xhigh · GPT-5.6 Luna·max · DeepSeek V4 Pro 0813 build·high · GLM-5.3·high · Claude Sonnet 5·max |
| MCP Atlas (table) openai | MCP Atlas; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| MCP-Atlas public (GLM-5 card) zhipu | MCP-Atlas 500-task public set think mode, 10 min timeout, judge Gemini 3 Pro; GLM-5 card; read 2026-10-19 ✓ | xmodel | GLM-5·think · GLM-4.7·think · GPT-5.2·xhigh |
| MCP-Atlas public set (GLM-5.2 card) zhipu | MCP-Atlas 500-task public set, think mode, 10 min timeout, judge Gemini-3.0-Pro; table value 76.8 (chart shows 77.0); read 2026-10-19 ✓ | xmodel | GLM-5.1·think · Claude Opus 4.8·max · GPT-5.5·xhigh · Gemini 3.1 Pro·high |
| mcptox github-mcptox | — | xmodel | Claude Opus 5·high · Claude Opus 4.8·nothink · Claude Sonnet 5·high |
| MDArena (50 molecular-dynamics tasks), single-run Pass@1 arxiv-2608.02642 | MDArena (arXiv 2608.02642, 31 Jul 2026): 50 tasks (29 systems, 14 protocols), each model-harness pair evaluated once per task (Pass@1), 900 s agent wall-clock default; score = Strict-Pass@1 % (Correctness and Process rewards are other columns); cost = mean recorded API cost per task (USD, Table 1); read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.5·medium |
| MedChem (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Sonnet 5·max · Claude Fable 5.1·max · Gemini 3.1 Pro·high · GPT-6 Astra·max |
| MedChemBench (OpenAI GPT-5.6 post chart) openai | MedChemBench, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| MedChemBench (OpenAI GPT-6 Astra post chart) openai | MedChemBench; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| MediCrab Lab notes #2: 24 planted errors in one pediatric septic-shock trial medicrab | same prompt, no web search, 2-3 runs each, graded blind; score = mean errors found of 24; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·medium · GPT-6 Astra·high · GPT-6.1 Sol·high · Gemini 4 Argon·high · Claude Sonnet 5.5·medium · Claude Haiku 5.5·high · GPT-6 Luna·medium · Gemini 4 Argon·medium |
| Melvynx medium duo github-melvynx | Claude Code · 2 apps, Fable 5 vs Sonnet 5 medium, cost only ✓ | xmodel | Claude Fable 5·medium · Claude Sonnet 5·medium |
| Melvynx xhigh trio github-melvynx | Claude Code · 3 apps, Fable 5 vs Sonnet 5 xhigh, same batch, verified runs, cost only ✓ | xmodel | Claude Fable 5·xhigh · Claude Sonnet 5·xhigh |
| Melvynx/benchmarks: web-app build, Claude Code, Opus 5.5 vs Sonnet 5.5 medium, 2 tasks without cursor continuation github-melvynx | Melvynx/benchmarks runs 2026-09-28; harness cc (Claude Code); tasks timezone-checker + video-generation summed; cost = accounting.json costUsd (harness cost, list basis); no quality score in repo (harness-verified only); read 2026-10-18 ✓ | xmodel | Claude Sonnet 5.5·medium · GPT-6 Astra·xhigh · GPT-6 Astra·medium · GPT-6 Sol·medium · GPT-6 Sol·xhigh |
| merc-bench hard tier T13 (per-task, adopted as core at 5e09655) github-merc-bench | github.com/sirneb/merc-bench, task T13, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓ | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·xhigh · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·low · Claude Opus 5.5·max · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·max · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·xhigh |
| merc-bench hard tier T14 (per-task, adopted as core at 5e09655) github-merc-bench | github.com/sirneb/merc-bench, task T14, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓ | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Claude Haiku 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Haiku 5.5·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·xhigh · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·low · Claude Opus 5.5·max · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·max · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·max · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·xhigh |
| merc-bench hard tier T15 (per-task, adopted as core at 5e09655) github-merc-bench | github.com/sirneb/merc-bench, task T15, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓ | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Fable 5·xhigh · Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Fable 5.1·medium · Claude Fable 5.1·xhigh · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Opus 4.8·high · Claude Opus 4.8·low · Claude Opus 4.8·xhigh · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·low · Claude Opus 5.5·max · Claude Opus 5.5·medium · Claude Opus 5.5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · Claude Sonnet 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·max · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·xhigh |
| Mercor · AdvancedIF mercor.com | Mercor, AdvancedIF: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Claude Fable 5.1·max · DeepSeek V4.1 Flash·max · Claude Opus 5.5·max · Claude Fable 5.1·high |
| Mercor · BrowseComp Extended mercor.com | Mercor, BrowseComp Extended: the public benchmark extended with new expert-built Mercor tasks (100 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Fable 5.1·max · GPT-6 Sol·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash·max |
| Mercor · CharXiv mercor.com | Mercor, CharXiv: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high |
| Mercor · CharXiv Extended mercor.com | Mercor, CharXiv Extended: the public benchmark extended with new expert-built Mercor tasks (150 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | GPT-6 Astra·xhigh · GPT-6.1 Sol·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max |
| Mercor · CVDP mercor.com | Mercor, CVDP: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Opus 5·max · Claude Sonnet 5.5·max · Grok 4.6·xhigh · GPT-5.6 Sol·max |
| Mercor · DeepResearch Bench II mercor.com | Mercor, DeepResearch Bench II: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5·high · Claude Fable 5.1·max · Claude Sonnet 4.6·high · Claude Opus 5.5·max · DeepSeek V4.1 Flash·max |
| Mercor · DeepSWE v1.1 mercor.com | Mercor, DeepSWE v1.1: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · GPT-6.1 Sol·max · GPT-6 Astra·max · DeepSeek V4.1 Flash·max · Claude Opus 5·max · Gemini 3.8 Flash·high |
| Mercor · FrontierSWE v2 mercor.com | Mercor, FrontierSWE v2: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Fable 5.1·high · GPT-6 Astra·max · Claude Opus 5·high · GPT-5.6 Sol·max · GLM-5.3·max |
| Mercor · GDPval mercor.com | Mercor, GDPval: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · DeepSeek V4.1 Flash·max · GPT-6 Astra·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max · Grok 4.6·xhigh · Claude Opus 5·max |
| Mercor · GDPval Extended mercor.com | Mercor, GDPval Extended: the public benchmark extended with new expert-built Mercor tasks (132 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · Claude Haiku 5.5·max · Claude Fable 5·max · GPT-6 Astra·max · Claude Opus 5·max |
| Mercor · Harvey LAB mercor.com | Mercor, Harvey LAB: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5·max · Claude Fable 5.1·max · Kimi K3·max · Claude Fable 5·max |
| Mercor · HLE mercor.com | Mercor, HLE: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · GPT-6 Astra·xhigh · Claude Fable 5.1·max · GPT-6.1 Sol·max · Claude Opus 5·max · Claude Fable 5·high |
| Mercor · HLE Extended mercor.com | Mercor, HLE Extended: the public benchmark extended with new expert-built Mercor tasks (447 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | GPT-5.6 Sol·max · GPT-5.6 Terra·max · Claude Opus 5·high · Claude Fable 5·high · GPT-5.6 Luna·max |
| Mercor · Long Context Reasoning (AA-LCR v1.1) mercor.com | Mercor, Long Context Reasoning (AA-LCR v1.1): the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Fable 5.1·max · Claude Fable 5·max · GPT-5.6 Sol·max · Claude Opus 5.5·max · Kimi K3·max |
| Mercor · Long Context Reasoning (AA-LCR) Extended mercor.com | Mercor, Long Context Reasoning (AA-LCR) Extended: the public benchmark extended with new expert-built Mercor tasks (89 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | GPT-5.6 Sol·max · GLM-5.3·max · GPT-5.6 Terra·max · Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·high |
| Mercor · MedXpertQA MM mercor.com | Mercor, MedXpertQA MM: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · GPT-6.1 Sol·max · Gemini 3.8 Flash·high · Claude Fable 5.1·max |
| Mercor · MedXpertQA MM Extended mercor.com | Mercor, MedXpertQA MM Extended: the public benchmark extended with new expert-built Mercor tasks (51 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · GPT-6 Sol·max · Claude Fable 5.1·high · GPT-5.6 Sol·max · GPT-6.1 Sol·max |
| Mercor · MMLU-Pro mercor.com | Mercor, MMLU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Opus 5·max · GPT-6 Astra·max · Gemini 3.1 Pro·high · Claude Fable 5.1·high · GPT-6 Astra·xhigh |
| Mercor · MMMU-Pro mercor.com | Mercor, MMMU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | GPT-6 Astra·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · GPT-6 Astra·xhigh · Gemini 3.7 Flash·high · Gemini 3.5 Flash·high |
| Mercor · MMMU-Pro Extended mercor.com | Mercor, MMMU-Pro Extended: the public benchmark extended with new expert-built Mercor tasks (440 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · GPT-5.6 Sol·max · GPT-6 Sol·max · Claude Opus 5.5·max · Claude Fable 5.1·max |
| Mercor · ProgramBench mercor.com | Mercor, ProgramBench: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·high · GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Opus 5·high · GPT-6 Sol·max |
| Mercor · SciCode mercor.com | Mercor, SciCode: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Sonnet 5.5·max · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| Mercor · SUPERChem mercor.com | Mercor, SUPERChem: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | GPT-6 Astra·xhigh · Claude Opus 5.5·max · GPT-6.1 Sol·max · Claude Sonnet 5.5·max · GPT-5.6 Sol·max · Claude Opus 5·max · Claude Fable 5.1·max · Gemini 3.7 Flash·high |
| Mercor · SWE Atlas - Codebase QnA mercor.com | Mercor, SWE Atlas - Codebase QnA: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | DeepSeek V4.1 Flash·max · Kimi K3·max · Claude Opus 5·max · Claude Sonnet 5.5·max · Claude Opus 5.5·max |
| Mercor · SWE Atlas - Codebase QnA Extended mercor.com | Mercor, SWE Atlas - Codebase QnA Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · DeepSeek V4.1 Flash·max |
| Mercor · SWE-bench Multilingual mercor.com | Mercor, SWE-bench Multilingual: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | DeepSeek V4.1 Flash·max · DeepSeek V4 Pro 0813 build·max · Claude Opus 5·max · Claude Opus 5.5·max |
| Mercor · SWE-bench Verified mercor.com | Mercor, SWE-bench Verified: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 5·max |
| Mercor · SWE-bench Verified Extended mercor.com | Mercor, SWE-bench Verified Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · Claude Opus 5·max · Claude Fable 5·max · GPT-5.6 Terra·max · GPT-5.6 Luna·max |
| Mercor · Terminal-Bench 2.1 Extended mercor.com | Mercor, Terminal-Bench 2.1 Extended: the public benchmark extended with new expert-built Mercor tasks (99 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 5.5·max · GPT-6.1 Sol·max · Grok 4.6·xhigh · Claude Sonnet 5.5·max · Gemini 3.8 Flash·high · GPT-6 Sol·max |
| Mercor APEX-Accounting mercor.com | Mercor's own board, APEX-Accounting, harness loop_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Fable 5.2·max · Gemini 4 Argon·high · Claude Fable 5.1·max · Claude Opus 5.5·max · GPT-6 Astra·max · Claude Sonnet 5.5·max · Claude Opus 5·max · GPT-6.1 Sol·max · Claude Haiku 5.5·max · GPT-5.6 Sol·max · Claude Fable 5·max · Grok 4.6·xhigh · GPT-5.6 Terra·max · GPT-6 Sol·max · GLM-5.3·max · Gemini 3.8 Flash·high · Grok 4.5·high · Gemini 3.7 Flash·high · Kimi K3·max · Claude Opus 4.7·max · GPT-5.5·xhigh · Claude Sonnet 5·max · Grok 4.7·xhigh · GPT-5.6 Luna·max · GPT-6 Luna·max · GLM-5.3-Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.6 Flash·high · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Kimi K2.7 Code·solo · Claude Opus 4.8·max · Gemini 3.1 Pro·high · GPT-5.4·xhigh · Claude Opus 5.5·medium |
| Mercor APEX-Agents mercor.com | Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Fable 5.2·max · Claude Fable 5.1·max · Gemini 3.7 Flash·high · Claude Opus 5·max · GPT-6 Astra·max · Grok 4.6·xhigh · Gemini 3.8 Flash·high · Claude Fable 5·max · GPT-5.6 Terra·max · GLM-5.3·max · Grok 4.5·high · GPT-5.5·xhigh · Claude Sonnet 5·max · GLM-5.3-Flash·max · GPT-5.4·xhigh · GPT-5.6 Sol·max · Kimi K3·max · Claude Opus 4.7·max · Claude Opus 4.8·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · Claude Fable 5.1·high · Gemini 3.1 Pro·high · Kimi K2.7 Code·solo · Claude Sonnet 4.6·high · Grok 4.7·xhigh · GPT-5.6 Luna·max · Claude Opus 5.5·medium · Claude Opus 5.5·max · GPT-6 Luna·max · GPT-6 Sol·max · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·high · Qwen3.8-27B·xhigh · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Gemini 4 Argon·high · Claude Haiku 5.5·max · DeepSeek V4.1 Flash 1017 build·max |
| Mercor APEX-SWE (inspect) mercor.com | Mercor's own board, APEX-SWE, harness inspect · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·high · Claude Sonnet 5·high · Grok 4.5·high · GPT-5.6 Sol·xhigh · Gemini 3.6 Flash·high |
| Mercor APEX-v1 (The AI Productivity Index) mercor.com | Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Gemini 3 Pro·high · Gemini 3 Flash·high · GPT-5.4·high · GPT-5.3-Codex·high · Gemini 3.1 Pro·high · Claude Sonnet 4.6·high · Claude Fable 5·max · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Opus 4.8·max · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · GPT-5.6 Terra·max · Kimi K2.7 Code·solo · Kimi K3·max · Claude Opus 5·xhigh · GPT-6 Astra·xhigh · GLM-5.3·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Fable 5.1·max · Grok 4.6·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·max · DeepSeek V4.1 Flash·max · GLM-5.3-Flash·max · Gemini 3.5 Flash-Lite·high · GPT-5.4·xhigh · GPT-5.5·xhigh · Grok 4.5·high · Gemini 3.7 Flash·high · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Opus 5.5·max · GPT-6 Sol·max · Qwen3.8-27B·xhigh · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Haiku 5.5·max · GPT-5.2·high · Gemini 4 Argon·high · DeepSeek V4.1 Flash 1017 build·max |
| Mercor open-source · GeneBench-Pro mercor.com | Mercor open-source evaluation, effort max (board field); read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·max · GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Opus 5.5·max |
| Mercor open-source · Terminal-Bench 4.0 mercor.com | Mercor open-source evaluation, effort max (board field); read 2026-10-15 ✓ | xmodel | GPT-6.1 Sol·max · Claude Opus 5.5·max · GPT-6 Astra·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max |
| mercor-apexacct-mean mercor.com | — | xmodel | Claude Fable 5.2·max · Gemini 4 Argon·high · Claude Fable 5.1·max · Claude Opus 5.5·max · GPT-6 Astra·max · Claude Sonnet 5.5·max · Claude Opus 5·max · GPT-6.1 Sol·max · Claude Haiku 5.5·max · GPT-5.6 Sol·max · Claude Fable 5·max · Grok 4.6·xhigh · GPT-5.6 Terra·max · GPT-6 Sol·max · GLM-5.3·max · Gemini 3.8 Flash·high · Grok 4.5·high · Gemini 3.7 Flash·high · Kimi K3·max · Claude Opus 4.7·max · GPT-5.5·xhigh · Claude Sonnet 5·max · Grok 4.7·xhigh · GPT-5.6 Luna·max · GPT-6 Luna·max · GLM-5.3-Flash·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.6 Flash·high · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Kimi K2.7 Code·solo · Claude Opus 4.8·max · Gemini 3.1 Pro·high · GPT-5.4·xhigh · Claude Opus 5.5·medium |
| mercor-apexv1-mean mercor.com | — | xmodel | Gemini 3 Pro·high · Gemini 3 Flash·high · GPT-5.4·high · GPT-5.3-Codex·high · Gemini 3.1 Pro·high · Claude Sonnet 4.6·high · Claude Fable 5·max · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Opus 4.8·max · Gemini 3.5 Flash·high · Gemini 3.6 Flash·high · GPT-5.6 Terra·max · Kimi K2.7 Code·solo · Kimi K3·max · Claude Opus 5·xhigh · GPT-6 Astra·xhigh · GLM-5.3·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Claude Fable 5.1·max · Grok 4.6·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·max · DeepSeek V4.1 Flash·max · GLM-5.3-Flash·max · Gemini 3.5 Flash-Lite·high · GPT-5.4·xhigh · GPT-5.5·xhigh · Grok 4.5·high · Gemini 3.7 Flash·high · Grok 4.7·xhigh · GPT-6 Luna·max · Claude Opus 5.5·max · GPT-6 Sol·max · Qwen3.8-27B·xhigh · Claude Sonnet 5.5·max · GPT-6.1 Sol·max · Claude Haiku 5.5·max · GPT-5.2·high · Gemini 4 Argon·high · DeepSeek V4.1 Flash 1017 build·max |
| messydocs-docexact github-messydocsbench | — | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| messydocs-docexact-contracts github-messydocsbench | — | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| messydocs-docexact-invoices github-messydocsbench | — | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| messydocs-docexact-receipts github-messydocsbench | — | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| messydocs-docexact-sroie github-messydocsbench | — | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| messydocs-docexact-statements github-messydocsbench | — | xmodel | Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| messydocs-docexact-w2 github-messydocsbench | — | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| Metacognition study: GPQA Diamond accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: GPQA Diamond (N=198), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: HLE accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: HLE (N=2,500), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: LiveBench accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LiveBench (N=400), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: LiveCodeBench v6 accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LiveCodeBench v6 (N=1,055), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: LiveMathBench accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LiveMathBench (N=240), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: LongBench v2 accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: LongBench v2 (N=503), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: MATH-500 accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MATH-500 (N=500), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: MathVision accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MathVision (N=3,040), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: MMLU-Pro accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MMLU-Pro (N=12,032), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: MMMU-Pro accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MMMU-Pro (N=3,460), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: MuSR accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: MuSR (N=756), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: OlympiadBench accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: OlympiadBench (N=6,728), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: SimpleQA accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: SimpleQA (N=4,326), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Metacognition study: ZebraLogic accuracy (single attempt, strict JSON output) metacog-2609.34864 | Metacognition-limits study (Han, Yang, Li; Microsoft Research Asia authors) Table 3: ZebraLogic (N=1,000), 4 models x 15 benchmarks x 38,238 items, single attempt, strict JSON output contract (format failures count as wrong), answer-time/pre/post confidence elicitation; judges gpt-5-mini medium for HLE/LiveMathBench else official scorers; own re-measurement of a public benchmark (separate group from public-board rows); arXiv 2609.34864 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · Grok 4.6·low |
| Mideind Icelandic LLM leaderboard, Case Government (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_case_marking@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Completion (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_cloze@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Famous Phrases (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_phrases@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Flora & Fauna (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_nature_translations@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Generative Quality (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark generative_quality@v1 primary metric quality_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Icelandic GEC (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_gec@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Icelandic WikiQA (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_wikiqa@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Meanings (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_meanings@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Song Lyrics (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_song_lyrics@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mideind Icelandic LLM leaderboard, Textile Terms (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_textiles@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·high · Gemini 4 Argon·high · GPT-6 Astra·high · Claude Fable 5.1·high · GPT-6 Astra·low · Claude Opus 5·high · GPT-6.1 Sol·high · GPT-6.1 Sol·low · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-6 Sol·high · GPT-5.6 Sol·high · Gemini 3.7 Flash·high · Claude Opus 4.8·high · Gemini 3.8 Flash·high · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Gemini 3.5 Flash·high · Claude Sonnet 5.5·high · GPT-6 Sol·nothink · GPT-5.6 Sol·nothink · GPT-6 Luna·high · Claude Sonnet 5·high · GPT-5.6 Luna·high · GPT-5.6 Terra·high · Claude Haiku 5.5·high · Gemini 3.5 Flash-Lite·high · GPT-5.6 Terra·nothink · GPT-6 Luna·nothink · GPT-5.2·high · GPT-5.6 Luna·nothink · GPT-5.2·nothink · Claude Haiku 4.5·nothink · GPT-6.1 Terra·high · Qwen3.8-27B·xhigh |
| Mine Odyssey: Minecraft spatial navigation (success rate, 180 tasks) arxiv-2610.11328 | 180 navigation tasks (125 outdoor/55 indoor), 500 decisions or 6 h; Opus 5 at 4 efforts plus Opus 5.5 max; Table 1 of arXiv 2610.11328; read 2026-10-15 ✓ | xmodel | GPT-6 Astra·medium · Claude Opus 5.5·max · Claude Opus 5·max · Claude Opus 5·high · Claude Opus 5·medium · Claude Opus 5·low · GPT-5.6 Sol·max · Gemini 3.8 Flash·high · Grok 4.6·xhigh · DeepSeek V4.1 Flash·max · GLM-5.3-Flash·max |
| MineBench voxel arena minebench | minebench.ai live API · voxel builds, 15 prompts, Elo arena read 29 Sep, measured cost for 8 of 12 Claude models ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · Gemini 4 Argon·high · Grok 4.7·xhigh · DeepSeek V4.1 Flash·max · Grok 4.6·xhigh · Gemini 3.8 Flash·high · GLM-5.3-Flash·max · Gemini 3.7 Flash·high · GLM-5.3·max |
| minilab: self-repo code review, Sonnet 5.5 (medium/high) vs Opus 5.5 (medium) note-minilab | note.com/minilab/n/n7ccd7f6a6421; author's own notectl CLI repo, 2 questions, 1 run per setting, Claude Code; cost = /usage session total (Q1+Q2) at list price, approximate (article uses 約); effort-from-config: "Sonnet 5.5のmedium、Sonnet 5.5のhigh、Opus 5.5。Opusのeffortは確認して、mediumでした"; read 2026-10-01 ✓ | xmodel | Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high |
| MMLU-Pro (GLM-4.7 card) zhipu | MMLU-Pro; GLM-4.7 card 2026-01-29; settings: default settings temp 1.0 top-p 0.95 131072; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| MMMU Pro (no tools) (table) openai | MMMU Pro (no tools); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| MMMU Pro (with tools) (table) openai | MMMU Pro (with tools); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| model-routing-eval: 3 fromager ticket tasks (r01t-r03t) github-model-routing-eval | andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['r01t_fromager_distinfo_ticket', 'r02t_fromager_age_filter_ticket', 'r03t_fromager_dep_chain_ticket']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·low · Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| model-routing-eval: 8 small coding tasks (t04-t12 subset) github-model-routing-eval | andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['t04_lru_cache', 't05_cli_json_flag', 't06_log_parser', 't07_mutable_default', 't08_timezone_bug', 't09_thread_safety', 't11_async_migration', 't12_needle_bug']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·low · Claude Sonnet 5·medium |
| moraru-opus5-aa-coding-agent-index gabrielmoraru.com | — | xmodel | Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| moraru-opus5-cursorbench gabrielmoraru.com | — | xmodel | Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| moraru-opus5-frontier-bench-v01 gabrielmoraru.com | — | xmodel | Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| morinokenjin 6th comparison: lesson slides (photosynthesis; template) zenn-morinokenjin | claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| morinokenjin 6th comparison: presentation slides (new service) zenn-morinokenjin | claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| morinokenjin 6th comparison: report slides (library usage) zenn-morinokenjin | claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender animation judged by Gemini 3.8 Flash (n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender mechanical watch judged by Gemini 3.8 Flash (n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender still life judged by Gemini 3.8 Flash (n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| Morphology (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Fable 5.1·max · Claude Opus 5.5·max · Gemini 3.1 Pro·high · GPT-6 Astra·max |
| mpgone-5-task-battery mpgone | — | xmodel | Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·max · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·max · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·max |
| mpgone-benchone-v2-2 mpgone | — | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·medium · Claude Fable 5.2·medium · Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium · Claude Opus 5·medium |
| mpgone-benchone-v2-2-coding mpgone | — | xmodel | Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium · Claude Opus 5·medium |
| mpgone-benchone-v2-2-data-math mpgone | — | xmodel | Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium · Claude Opus 5·medium |
| mpgone-benchone-v2-2-facts-honesty mpgone | — | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·medium · Claude Fable 5.2·medium · Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium · Claude Opus 5·medium |
| mpgone-benchone-v2-2-long-context mpgone | — | xmodel | Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium |
| mpgone-benchone-v3-1 mpgone | — | xmodel | Claude Opus 5.5·medium · Claude Fable 5.2·medium · Claude Sonnet 5.5·medium · Claude Opus 5·medium · Claude Fable 5.1·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium |
| mpgone-benchone-v3-1-agents mpgone | — | xmodel | Claude Fable 5.1·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium |
| mpgone-benchone-v3-1-coding mpgone | — | xmodel | Claude Fable 5.1·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium |
| mpgone-benchone-v3-1-data-math mpgone | — | xmodel | Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium |
| mpgone-benchone-v3-1-instructions-writing mpgone | — | xmodel | Claude Opus 5.5·medium · Claude Fable 5.2·medium · Claude Sonnet 5.5·medium · Claude Opus 5·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium |
| mpgone-benchone-v3-1-long-context mpgone | — | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·medium |
| mpgone-benchone-v3-1-reasoning mpgone | — | xmodel | Claude Opus 5.5·medium · Claude Fable 5.2·medium · Claude Sonnet 5.5·medium · Claude Opus 5·medium · Claude Fable 5.1·medium · Claude Haiku 5.5·medium · Claude Sonnet 5·medium |
| mre-uxd-u01 github-model-routing-eval | — | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · GPT-6 Astra·low · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6.1 Sol·low · GPT-6 Astra·medium · GPT-6.1 Sol·medium |
| mre-uxd-u01-pass github-model-routing-eval | — | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · GPT-6 Astra·low · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6.1 Sol·low · GPT-6 Astra·medium · GPT-6.1 Sol·medium |
| mre-uxdws-u02 github-model-routing-eval | — | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6.1 Sol·low · GPT-6.1 Sol·medium |
| MTEB(LLM) overall score (37 tasks) and cost per pass, LLMs prompted as embedders embedder-2608.12875 | MTEB(LLM) 37 tasks (8 classification, 9 clustering, 10 STS, 4 pair-classification, 6 retrieval) run by prompting LLMs; metric mean of category means 0-100; cost = USD for one full pass from API token usage x public rates (March/June 2026), reasoning tokens at output rate; per-category columns are the same runs; Gemini 3 Flash 'low' ablation covers retrieval only (not a group: mean not printed); arXiv 2608.12875 COLM 2026 Tables 1/17; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Gemini 3 Flash·high |
| NanoGPT (OpenAI GPT-5.6 post chart) openai | NanoGPT, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| Negotiation deal-closure, Control condition (Table 2, mean over 240/300/360 s) arxiv-2601.13206 | arXiv 2601.13206v2 Table 2, deal-closure %, simulated negotiation, N=300 per condition; provider-default settings; GPT-5.1-chat-latest called 2025-11-22 (Table A.2); control and time-aware are two prompts (two sets of runs); read 2026-10-20 ✓ | xmodel | GPT-5.1 Chat (latest)·solo · GPT-5.1·medium |
| Negotiation deal-closure, Time-Aware condition (Table 2, mean over 240/300/360 s) arxiv-2601.13206 | arXiv 2601.13206v2 Table 2, deal-closure %, simulated negotiation, N=300 per condition; provider-default settings; GPT-5.1-chat-latest called 2025-11-22 (Table A.2); control and time-aware are two prompts (two sets of runs); read 2026-10-20 ✓ | xmodel | GPT-5.1 Chat (latest)·solo · GPT-5.1·medium |
| neocognition-apprentice-bench-api neocognition | — | xmodel | Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Grok 4.6·xhigh · Grok 4.5·high |
| neocognition-apprentice-bench-cua neocognition | — | xmodel | Gemini 3.8 Flash·high · Gemini 3.7 Flash·high |
| NeutronGym McStasBench seen tasks (of 16), one-shot arxiv-2610.03631 | 16 seen McStasBench tasks, one episode each, temperature 0; Table 1 of arXiv 2610.03631; read 2026-10-15 ✓ | xmodel | Claude Haiku 4.5·nothink · Gemini 3.6 Flash·medium · Gemini 3.5 Flash-Lite·minimal |
| Next.js Agent Evals (Vercel), base run nextjs-evals | 31 Next.js agent evals, 4 runs each (earlyExit), Claude Code via AI Gateway, no AGENTS.md; https://nextjs.org/evals, github.com/vercel/next-evals-oss@38cb8ad, agent-results.json metadata.experiments[].baseSuccessRate/avgCostUsd; read 2026-10-01 ✓ | xmodel | Claude Fable 5·high · Claude Fable 5.1·high · Claude Opus 5.5·high · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-6 Astra·high · GPT-6 Luna·high · GPT-6 Sol·high · Kimi K2.7 Code·solo |
| Next.js Agent Evals (Vercel), with AGENTS.md nextjs-evals | Same 31 evals/4 runs/Claude Code setup as nextjsevals-base, but with an AGENTS.md/CLAUDE.md file telling the agent the Next.js version has breaking changes; score = docsSuccessRate (board); cost = mean of per-eval costUsd in agent-results.json['results'][model] (computed-by-us, since the board's avgCostUsd field is the base-run figure, not the AGENTS.md-run figure — the per-eval array's own mean, 0.2456 for Opus 5.5, differs from the base 0.234, confirming the two are different runs); read 2026-10-01 ✓ | xmodel | Claude Fable 5·high · Claude Fable 5.1·high · Claude Opus 5.5·high · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-6 Astra·high · GPT-6 Luna·high · GPT-6 Sol·high · Kimi K2.7 Code·solo |
| nipun-word-break-micro nipunbatra.github.io | — | xmodel | Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.1 Flash-Lite·minimal · Gemini 3.5 Flash·high |
| NL2Repo (GLM-5.2 card) zhipu | NL2Repo temperature=1.0 top_p=1.0 max_new_tokens=48k 400k context, rule+LLM anti-hack; GLM-5.2 card; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · Gemini 3.1 Pro·high |
| nnakapa-o47o48-t2 zenn-nnakapa | — | xmodel | Claude Opus 4.7·xhigh · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh |
| nnakapa-o5s5-t2 zenn-nnakapa | — | xmodel | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh |
| nnakapa-s5cc-t2 zenn-nnakapa | — | xmodel | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| nnakapa-s5cop-t2 zenn-nnakapa | — | xmodel | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh |
| NonoBench nonobench.com | NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓ | xmodel | Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Gemini 4 Argon·low · Gemini 4 Argon·medium · Gemini 4 Argon·high · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · Grok 4.7·low · Grok 4.7·medium · Grok 4.7·high · Grok 4.7·xhigh · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · GLM-5.3-Flash·low · GLM-5.3-Flash·high · GLM-5.3-Flash·max · GLM-5.3·low · GLM-5.3·high · GLM-5.3·max · Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Kimi K3·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.4·low · GPT-5.4·high · GPT-5.4·xhigh · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high · Gemini 3 Flash·high · GPT-5.2·low · GPT-5.2·high · GPT-5.2·xhigh · Gemini 3 Pro·high · Grok 4.1 Fast·think · GLM-4.7·think |
| note AI実験室 #1 arithmetic_parser (165 hidden pytest tests) note-sane_mink364 | Claude Code CLI (subscription); hidden pytest; API-equivalent cost from tokens x list price; Haiku 5.5 low/medium vs Opus 5.5 medium; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·medium · Claude Opus 5.5·medium |
| note AI実験室 #1 duration_parse (29 hidden pytest tests) note-sane_mink364 | Claude Code CLI (subscription); hidden pytest; API-equivalent cost from tokens x list price; Haiku 5.5 low/medium vs Opus 5.5 medium; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·medium · Claude Opus 5.5·medium |
| note AI実験室 #2-#4 spec contradiction detection, hard (28 req x 3 specs x 5 runs) note-sane_mink364 | Claude Code CLI; 3 specs x 5 runs = 15 per condition; F1 on contradiction IDs (auto-scored; code fences stripped for all); cost API-equivalent from tokens x price; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·medium · Claude Haiku 5.5·max · Claude Opus 4.8·high |
| NSMQ Riddles offline exact match (156 riddles) arxiv-2605.07051 | NSMQ Riddles (Ghana quiz science/maths riddles), 2019 set, 156 riddles, offline = all clues at once, one attempt, no tools; temperature 1; open models via Together AI API, closed via LangChain wrapper for the OpenAI API (paper Sec.4); low/high reasoning effort; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high |
| NSMQ Riddles real-time proxy exact match (156 riddles) arxiv-2605.07051 | NSMQ Riddles 2019 real-time proxy: riddle clues passed in 7-word chunks, model answers and states sure/not sure; 156 riddles; temperature 1; same models/providers as offline; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high |
| nurb-bit-block github-nurb | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·xhigh · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · GPT-5.6 Terra·high · GPT-5.6 Luna·medium · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·medium · GPT-5.6 Luna·max · Grok 4.6·high · GPT-5.6 Luna·high · GPT-5.6 Terra·low · GPT-5.6 Luna·low |
| nurb-bundle-holder github-nurb | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · GPT-5.6 Sol·high · GPT-5.6 Terra·max · GPT-5.6 Terra·xhigh · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Claude Sonnet 5·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·medium · GPT-5.6 Sol·medium · GPT-5.6 Luna·high · GPT-5.6 Sol·max · GPT-5.6 Terra·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·low · GPT-5.6 Terra·high · GPT-5.6 Terra·low · GPT-5.6 Luna·low · Claude Sonnet 5·low |
| nurb-cable-clip github-nurb | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·medium · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·xhigh · Grok 4.6·high · Grok 4.6·xhigh · GPT-5.6 Terra·high · GPT-5.6 Luna·low · Claude Sonnet 5·medium |
| nurb-leg-cup github-nurb | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · GPT-5.6 Luna·high · GPT-5.6 Terra·high · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · GPT-5.6 Terra·xhigh · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·medium · Claude Opus 5·medium · Claude Opus 5·max · GPT-5.6 Terra·low · GPT-5.6 Luna·low |
| nurb-pole-rest github-nurb | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Opus 5·high · Claude Opus 5·low · Claude Opus 5·max · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Sonnet 5·high · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · GPT-5.6 Luna·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·max · GPT-5.6 Terra·medium · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·low · Claude Sonnet 5·low · GPT-5.6 Luna·low |
| nurb-valve-knob github-nurb | — | xmodel | Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Opus 5·high · Claude Opus 5·low · Claude Sonnet 5·high · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-5.6 Terra·xhigh · Grok 4.6·high · Grok 4.6·low · Grok 4.6·medium · Grok 4.6·xhigh · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Luna·high · GPT-5.6 Luna·max · GPT-5.6 Luna·xhigh · GPT-5.6 Sol·medium · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·medium · Claude Opus 5·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·low · GPT-5.6 Sol·high · GPT-5.6 Luna·low · GPT-5.6 Luna·medium |
| Nuxt agent evals (nuxt/nuxt-evals), Claude Code via AI Gateway nuxt/nuxt-evals | agent-results.json exportedAt 2026-09-22; 31 Nuxt agent evals, runs: 4, earlyExit: true; score is avgPassRate (fraction of the 4 runs passing, averaged over 31 evals), *100 for a percentage; cost is the gateway's avg cost USD per eval run; models ran on different dates (opus-5.5 2026-09-22, the rest 2026-08-27 to 09-05); read 2026-10-01 ✓ | xmodel | Claude Fable 5·high · Claude Fable 5.1·high · Claude Opus 4.8·high · Claude Opus 5·high · Claude Opus 5.5·medium · Claude Sonnet 5·high · Claude Sonnet 5.5·medium · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.6 Sol·xhigh · GPT-6 Astra·high · GPT-6 Sol·high · Kimi K2.7 Code·solo |
| nuxt-evals-success nuxt/nuxt-evals | — | xmodel | Claude Fable 5·high · Claude Fable 5.1·high · Claude Opus 4.8·high · Claude Opus 5·high · Claude Opus 5.5·medium · Claude Sonnet 5·high · Claude Sonnet 5.5·medium · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-6 Astra·high · GPT-6 Sol·high · Kimi K2.7 Code·solo |
| NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections) matsonj/eval-connections | results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓ | xmodel | GPT-6 Astra·low · Gemini 4 Argon·low · Grok 4.6·low · GPT-6 Sol·low · Gemini 3.8 Flash·low · Claude Opus 5.5·low · Gemini 3.1 Pro·low · GPT-5.6 Terra·low · Claude Fable 5·low · Grok 4.7·low · Gemini 3 Flash·minimal · Kimi K3·low · GPT-5.6 Sol·low · DeepSeek V4.1 Flash 1017 build·low · Claude Fable 5.1·low · Gemini 3.7 Flash·low · GLM-5.3-Flash·low · Claude Sonnet 5.5·low · Grok 4.5·low · Claude Opus 5·low · GPT-5.6 Luna·low · DeepSeek V4.1 Flash·low · Claude Sonnet 5·low · Gemini 3.6 Flash·minimal · DeepSeek V4 Flash preview·high · Gemini 3.5 Flash-Lite·minimal · Claude Haiku 5.5·low · Gemini 3.5 Flash·minimal · GPT-6 Luna·low · GLM-5.3·low · Qwen3.8-27B·low |
| oai-bio-basicsbench openai | — | xmodel | GPT-6 Astra·xhigh · GPT-6.1 Sol·xhigh · GPT-5.6 Sol·xhigh · GPT-6 Luna·xhigh · GPT-5.5·xhigh · GPT-5.4·xhigh · GPT-5.2·xhigh |
| oai-sc-exploitbench openai | — | xmodel | GPT-6 Sol·max · GPT-6 Luna·max |
| oai-sc-mentalhealthbench-emergent openai | — | xmodel | GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max |
| oai-sc-mentalhealthbench-high-acuity openai | — | xmodel | GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max |
| oai-sc-mentalhealthbench-non-acute openai | — | xmodel | GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max |
| oai-sc-mentalhealthbench-overall openai | — | xmodel | GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6.1 Terra·max |
| oai-t61-agents-last-exam-v1 openai | — | xmodel | Claude Opus 5.5·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-6.1 Terra·low · GPT-6.1 Terra·medium · GPT-6.1 Terra·high · GPT-6.1 Terra·xhigh · GPT-6.1 Terra·max |
| oai-t61-factuality-error-rate openai | — | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-6.1 Terra·low · GPT-6.1 Terra·medium · GPT-6.1 Terra·high · GPT-6.1 Terra·xhigh · GPT-6.1 Terra·max |
| oai-t61-mmmu-pro-vision openai | — | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-6.1 Terra·low · GPT-6.1 Terra·medium · GPT-6.1 Terra·high · GPT-6.1 Terra·xhigh · GPT-6.1 Terra·max |
| oai-t61-osworld-2-1-offline openai | — | xmodel | Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-6.1 Terra·low · GPT-6.1 Terra·medium · GPT-6.1 Terra·high · GPT-6.1 Terra·xhigh · GPT-6.1 Terra·max |
| oai-t61-terminal-bench-4-0 openai | — | xmodel | Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-5.6 Terra·max · GPT-6.1 Terra·low · GPT-6.1 Terra·medium · GPT-6.1 Terra·high · GPT-6.1 Terra·xhigh · GPT-6.1 Terra·max |
| ockbench-top200 ockbench | — | xmodel | Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · Claude Opus 5·low · GPT-5.5·medium · Gemini 3.1 Pro·medium · Claude Opus 4.8·xhigh · GPT-5.5·high · GPT-5.5·xhigh · Claude Opus 5·medium · Claude Opus 4.8·medium · Claude Opus 4.8·high · GPT-5.4·medium · GPT-5.4·high · Claude Opus 4.7·xhigh · Gemini 3.1 Pro·low · GPT-5.4·low · Claude Opus 4.8·max · GPT-5.5·low · Claude Opus 4.8·low · DeepSeek V4 Pro preview·high · GLM-5.2·high |
| OfficeQA (S55) anthropic | Sonnet 5.5 card p132 · 3 models at max, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5·max |
| OfficeQA Pro (S55) anthropic | Sonnet 5.5 card p132 · 133 q, 3 models at max, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5·max |
| OfficeQA Pro (table) openai | OfficeQA Pro; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAgents briefing-model factorial (control) github-openagents | Claude Code (coder-connect agent) · frozen historical reserve issue, control arm, Opus 5.5 vs Sonnet 5.5 × medium effort, n=4 reps each; score=pass-at-first/final-attempt%, cost=cumulative list-price USD; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| OpenAgents briefing-model factorial (treatment) github-openagents | Claude Code (coder-connect agent) · frozen historical reserve issue, treatment arm, Opus 5.5 vs Sonnet 5.5 × medium effort, n=4 reps each; score=pass-at-first/final-attempt%, cost=cumulative list-price USD; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium |
| OpenAI Codex-Spark launch chart · SWE-Bench Pro accuracy by reasoning effort openai-codex-spark-launch | OpenAI launch post 2026-02-12, vega chart SWE-Bench Pro (accuracy vs task duration; axis 'Reasoning effort'); harness / attempts / subset not stated on the page; read 2026-10-20 ✓ | xmodel | GPT-5.3-Codex·medium · GPT-5.3-Codex·high · GPT-5.3-Codex·xhigh |
| OpenAI MRCR v2 8-needle 128K-256K (table) openai | OpenAI MRCR v2 8-needle 128K-256K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAI MRCR v2 8-needle 16K-32K (table) openai | OpenAI MRCR v2 8-needle 16K-32K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAI MRCR v2 8-needle 256K-512K (table) openai | OpenAI MRCR v2 8-needle 256K-512K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAI MRCR v2 8-needle 32K-64K (table) openai | OpenAI MRCR v2 8-needle 32K-64K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAI MRCR v2 8-needle 4K-8K (table) openai | OpenAI MRCR v2 8-needle 4K-8K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAI MRCR v2 8-needle 512K-1M (table) openai | OpenAI MRCR v2 8-needle 512K-1M; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAI MRCR v2 8-needle 64K-128K (table) openai | OpenAI MRCR v2 8-needle 64K-128K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| OpenAI MRCR v2 8-needle 8K-16K (table) openai | OpenAI MRCR v2 8-needle 8K-16K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| openclaw-opus55-goal-build openclawdatabase | — | xmodel | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| OpenMC neutron-shielding single-shot script generation (v2): execution gate, L1-L5 zenodo-neutron-shielding | zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) Table §5.1 'the gate' (scripts that ran, of 15 total, L1-L5); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Sol·medium · Gemini 3.8 Flash·medium |
| OpenMC neutron-shielding single-shot script generation (v2): physics-correct, L2-L4 zenodo-neutron-shielding | zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) §5.2 (physics-correct of 9, L2-L4, the rungs with a scorable physics criterion); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · GPT-6 Sol·medium · Gemini 3.8 Flash·medium |
| OpenRouter GPQA-Diamond rolling-mean board (snapshot 2026-10-03) openrouter | reasoning-math/hard, OR-internal rolling-mean harness, auto-routing provider aggregate; ref openrouter.ai/benchmarks/gpqa-diamond; SNAPSHOT REPLACES prior orgpqa snapshot (2026-09-27); read 2026-10-05 ✓ | xmodel | Kimi K2 Thinking·solo · Kimi K2.7 Code·solo |
| OpenRouter tau2-bench-airline board (snapshot 2026-10-03) openrouter | agentic-tool/mid, tau2-bench airline tasks, OR-internal harness, auto-routing provider aggregate; ref openrouter.ai/benchmarks/tau2-bench-airline; SNAPSHOT REPLACES prior ortau2 snapshot (2026-09-27); read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K2 Thinking·solo |
| OpenRouter VGI-Bench (video general intelligence, 439 multiple-choice questions on long videos): accuracy and cost per question openrouter-vgi-bench | VGI-Bench public split 439 questions (by Seldon), run through OpenRouter default routing, one turn, full video + question, temperature 0, graded letter after 'Answer:'; listed only with >=395 answered; scores aggregate successful runs of the newest 90 days; board last run 2026-10-18 12:18 UTC; cost = average cost per question as printed; read 2026-10-20 ✓ | xmodel | GLM-5.3-Flash·max · Gemini 4 Argon·medium · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·medium · Gemini 3.5 Flash-Lite·minimal |
| OpenRUA: Claude Code driving a robot, LIBERO-10 (LIBERO-PRO) arxiv-2610.02459 | 200 trials per configuration across both perturbation cells, Claude Code at high effort; Table 11 of arXiv 2610.02459; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5·high · Claude Opus 5·high · Claude Fable 5.1·high · GPT-6 Astra·medium · GPT-6 Astra·high |
| OpenScore String Quartets (Legato camera subset) (OpenAI GPT-6 Astra post chart, non-code mode) openai | OpenScore String Quartets (Legato camera subset); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| openvl-base-avg22 arxiv-2610.26860 | — | xmodel | GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Astra·medium · GPT-6 Sol·medium · Kimi K3·max |
| openvl-codex-avg22 arxiv-2610.26860 | — | xmodel | GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max |
| openvl-ours-avg22 arxiv-2610.26860 | — | xmodel | GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Astra·medium · GPT-6 Sol·medium · Kimi K3·max |
| openvl-vs-avg22 arxiv-2610.26860 | — | xmodel | GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max |
| OS-Bench cross-platform migration, ClaudeCode, Opus 4.8 vs Sonnet 5 arxiv-2610.19368 | arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 91 Linux tasks; medium thinking effort; read 2026-10-18 ✓ | xmodel | Claude Opus 4.8·medium · Claude Sonnet 5·medium |
| OS-Bench cross-platform migration, SWE-agent, Opus 4.8 vs Sonnet 5 arxiv-2610.19368 | arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 91 Linux tasks; medium thinking effort; read 2026-10-18 ✓ | xmodel | GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Claude Opus 4.8·medium · Claude Sonnet 5·medium |
| OS-Bench cross-platform migration, Win-agent, Opus 4.8 vs Sonnet 5 arxiv-2610.19368 | arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 49 Windows tasks; medium thinking effort; read 2026-10-18 ✓ | xmodel | GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Claude Opus 4.8·medium |
| OSWorld 2.0 (105 tasks) in the CUAWright harness (authors' system) arxiv-2610.04116 | CUAWright (arXiv 2610.04116, 2 Oct 2026) Table 7: OSWorld 2.0, 105 tasks scored, CUAWright harness (the authors' own computer-use harness) for both models; score = partial score % (first metric of the paper's OSWorld table; exact passes 27 and 29 of 105 are another summary); cost = 'Current Standard cost' of the run converted to cost per task; published baselines of other harnesses in its Table 2 are restated from boards already held and are not rows; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.6 Sol·max |
| OSWorld 2.0 batch arxiv-2606.29537 | arXiv 2606.29537 · 108 workflows, 500 steps, batched tool calls, max ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·max · GPT-5.5·xhigh |
| OSWorld 2.0 code_mode=false (OpenAI GPT-5.6 post chart) openai | OSWorld 2.0, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max |
| OSWorld 2.0 offline (OpenAI GPT-6 Astra post chart) openai | OSWorld 2.0 offline; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| OSWorld 2.0 single arxiv-2606.29537 | arXiv 2606.29537 · 108 workflows, 500 steps, single tool call, max ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·max |
| OSWorld 2.0 single 150 steps osworld-v2-board | osworld-v2.xlang.ai official JSON · 108 tasks, 150-step budget, Opus 4.7 max, Sonnet 4.6 medium/max ✓ | xmodel | Claude Opus 4.7·max · GPT-5.5·xhigh |
| OSWorld 2.0 single 300 steps osworld-v2-board | osworld-v2.xlang.ai official JSON · 108 tasks, 300-step budget, Opus 4.7 max, Sonnet 4.6 medium/max ✓ | xmodel | Claude Opus 4.7·max · GPT-5.5·xhigh |
| OSWorld 2.0, offline set (OpenAI GPT-6 Sol and Luna post) openai | OSWorld 2.0, offline set; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6 Luna·low · GPT-6 Luna·medium · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| OSWorld 2.0, offline set (OpenAI GPT-6.1 Sol post) openai | OSWorld 2.0, offline set; OpenAI GPT-6.1 Sol post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; Wayback first capture; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max |
| OSWorld-Pro arxiv-2609.24890 | arXiv 2609.24890 · OSWorld harness, 250 turns, 4 Claude models all at max, tokens × OpenRouter list incl. cache ✓ | xmodel | Claude Opus 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · Claude Sonnet 5·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-5.6 Luna·max · Gemini 3.8 Flash·high · Kimi K3·max · Qwen3.8-27B·xhigh |
| OSWorld-Verified (OpenAI GPT-5.5 post chart) openai | OSWorld-Verified; OpenAI GPT-5.5 launch post chart per reasoning effort (none..xhigh); no cost on this chart (tokens/tool calls only); published 2026-04-23 (snapshot Wayback 20260423180657); read 2026-10-19 ✓ | xmodel | GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh |
| osworld2-offline-batch500-bin osworld-v2-board | — | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| osworld2o5-bin osworld-v2-board | — | xmodel | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max |
| osworld2o5a-bin osworld-v2-board | — | xmodel | Claude Opus 5·low · Claude Opus 5·medium · Claude Opus 5·high · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-5.6 Sol·max |
| osworld2s150-bin osworld-v2-board | — | xmodel | Claude Opus 4.7·max · GPT-5.5·xhigh |
| osworld2s300-bin osworld-v2-board | — | xmodel | Claude Opus 4.7·max · GPT-5.5·xhigh |
| PA100 rubric verification, verifier AAAJ, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier AAAJ; cost per full run; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| PA100 rubric verification, verifier Archipelago, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Archipelago; cost per full run; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| PA100 rubric verification, verifier Autorubric, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Autorubric; cost per full run; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| PA100 rubric verification, verifier Gandalf, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Gandalf; cost per full run; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5·high |
| Parallel Search Capability · with search parallel.ai | Parallel Search Capability Leaderboard, Search Intelligence Score = mean of DSQA, HLE and WISER (100 questions each) with Parallel Search; cost per task incl. estimated search/extract usage; Claude 5.x at max effort per the page notes (Haiku 5.5 effort unstated); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · Gemini 4 Argon·high · GPT-6.1 Sol·max · Claude Opus 5·max · GPT-6 Sol·max · Claude Sonnet 5·max · Grok 4.7·medium · GPT-6 Luna·max |
| Parallel Search Capability · without search parallel.ai | Same suites without the search tool (model alone); inference cost only; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · GPT-6.1 Sol·max · GPT-6 Sol·max · Claude Opus 5·max · Gemini 4 Argon·high · Claude Sonnet 5.5·max · GPT-6 Luna·max · Claude Sonnet 5·max · Grok 4.7·medium |
| ParseBench document-parsing leaderboard (run-llama/ParseBench) run-llama/ParseBench | leaderboard.csv at commit afb36bd8 (2026-09-29); Overall score (Tables/Charts/ContentFaithfulness/SemanticFormatting/VisualGrounding average); cost is USD-cents per page from the repo's own pricing table (not independently verified against token counts, so flagged list-price-cost); n (documents) and run count not given in the csv; read 2026-10-01 ✓ | xmodel | Claude Haiku 4.5·nothink · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · GPT-6.1 Sol·low · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·high · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.8 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Claude Opus 4.8·nothink · Claude Opus 5.5·high · Claude Opus 5.5·low · Claude Fable 5.1·high · Claude Sonnet 5.5·high · Gemini 3.1 Pro·low · Gemini 3.1 Flash-Lite·medium · Gemini 3.1 Flash-Lite·high · Gemini 3.5 Flash-Lite·high · Gemini 3.5 Flash-Lite·medium · GPT-5.4·nothink · GPT-5.5·medium · GPT-5.5·nothink · GPT-5.6 Terra·medium · GPT-5.6 Terra·low · GPT-5.6 Terra·nothink · GPT-5.6 Sol·high · GPT-5.6 Sol·nothink · GPT-5.6 Luna·max · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·high · GPT-5.6 Luna·medium · GPT-5.6 Luna·low · GPT-5.6 Luna·nothink · GPT-6 Astra·low · GPT-6 Sol·high · GPT-6 Sol·medium · GPT-6 Sol·nothink · GPT-6 Luna·max · GPT-6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Luna·medium · GPT-6 Luna·low · GPT-6 Luna·nothink · DeepSeek V4.1 Flash·high |
| patterneval-acc arxiv-2608.12781 | — | xmodel | Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink · Qwen3.5-122B-A10B·nothink · Qwen3.6-27B·nothink |
| pd-control arxiv-2608.02670 | — | xmodel | GPT-5.6 Sol·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high · Claude Sonnet 5·high · Claude Opus 4.8·high · Claude Fable 5·high · Claude Opus 5·high · Grok 4.5·high |
| pd-nisthigh arxiv-2608.02670 | — | xmodel | GPT-5.6 Sol·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high · Claude Sonnet 5·high · Claude Opus 4.8·high · Claude Fable 5·high · Claude Opus 5·high · Grok 4.5·high |
| pd-nonroot arxiv-2608.02670 | — | xmodel | GPT-5.6 Sol·high · GPT-5.6 Terra·high · GPT-5.6 Luna·high · Claude Sonnet 5·high · Claude Opus 4.8·high · Claude Fable 5·high · Claude Opus 5·high · Grok 4.5·high |
| pebench-private getcompound | — | xmodel | Claude Opus 5.5·medium · Claude Fable 5.1·high · GPT-6 Astra·low · GPT-6.1 Sol·low · GPT-6 Luna·medium |
| PeterTh llm-eval-local: LLM autoparallelization of 220 programs (Claude Code, medium) github-peterth-llm-eval-local | PeterTh/llm-eval-local analysis/tables/4d_all_models_score_vs_cost.csv; 220 programs per model-effort profile; slugs claude- | xmodel | GPT-6 Luna·medium · GPT-5.6 Luna·xhigh · Claude Haiku 5.5·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Claude Sonnet 5.5·medium · GPT-5.6 Terra·xhigh · GPT-6.1 Sol·xhigh · GPT-5.6 Sol·xhigh · GPT-6 Astra·medium · Claude Sonnet 5·medium · Claude Opus 5.5·medium · Claude Fable 5·medium · Claude Opus 5·medium · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Sol·low · GPT-5.6 Sol·medium |
| Phi-Bench (LLM infrastructure engineering, Full score), Claude Code, Opus 5 vs Sonnet 5 arxiv-2609.10226 | arXiv 2609.10226 v2 (15 Oct 2026); Claude Code; Table 2 Full score; one H20 GPU/task; read 2026-10-18 ✓ | xmodel | Claude Opus 5·max · Claude Sonnet 5·max · GPT-5.6 Sol·max |
| phi-Bench: file-system design and implementation, average pass rate arxiv-2608.00280 | Benchmarking LLMs on File System Design and Implementation (UIUC, arXiv 2608.00280, 4 Aug 2026) Figure 4 'Average' bars: pass rate % over six task types (BU/BI/PM/DE/PO/NF), 5 trials per task averaged; DIGITISED from the figure (src/img/phi_f4_zoom.png, pixel scale from the 60%/80% gridlines, +-1.5 points; script verify/phi_bars.py); Figure 10 labels give DeepSeek-V4-Flash 46 and Gemini-3.1-Pro 64 (different weighting); read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Claude Opus 4.7·max · GPT-5.2·xhigh |
| phyblas Tibetan-script reading web page (Vue), blind grading qiita-phyblas | claude -p headless --effort medium, 3 models concurrently, 1 run each; blind grading by Opus 5.5 at high; 100 Wiktionary test words; cost = total_cost_usd (API pricing); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium |
| PhysAI-Bench held-out decision accuracy (500 instances, model-specific frozen configuration) physai-2609.23695 | PhysAI-Bench (Ferrag et al.; physical AI / autonomous-network decisions) Table IV: held-out set of 500 decision instances, 3 prespecified runs, per-model few-shot count and temperature chosen on a 35-instance development set (config differs per row); development-set overall accuracy over all configurations is a tuning table, not taken; arXiv 2609.23695 (Sep 2026); read 2026-10-20 ✓ | xmodel | GPT-5.2 Chat (latest)·solo |
| PM-LLM-Benchmark v3, process-mining Q&A, judged by grok-4.7 (fit-alessandro-berti/pm-llm-benchmark3) fit-alessandro-berti/pm-llm-benchmark3 | leaderboard_grok-4.7.md at commit ab1371c (2026-09-30); 1-shot, 10-category score out of 60, LLM judge (grok-4.7, not a Claude judge); single run; no cost in this board; read 2026-10-01 ✓ | xmodel | GPT-6 Astra·xhigh · GPT-6.1 Sol·xhigh · GPT-6 Sol·xhigh · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·xhigh · GPT-6 Luna·xhigh · Claude Opus 5·high · Claude Fable 5·high · GPT-5.6 Sol·nothink · GPT-5.6 Luna·xhigh · GPT-5.4·xhigh · GPT-5.2·xhigh · GPT-5.6 Terra·nothink · Gemini 3.8 Flash·high · GPT-5.1·high · Gemini 3.7 Flash·high |
| pokebench.tv: Pokemon Red speedrun to Brock, turns/time/cost pokebench | fresh-save Pokemon Red, screenshot+RAM state per turn, budget 1000 turns, Anthropic API list-price cost; https://pokebench.tv/runs.json, src/pokebench_runs.json; read 2026-10-05 ✓ | xmodel | GPT-6.1 Sol·high · Claude Opus 5.5·high · Gemini 3.8 Flash·high · Grok 4.6·high · Grok 4.7·high · Claude Fable 5.1·high · GPT-6 Sol·high · GPT-6 Luna·high · DeepSeek V4.1 Flash·high · GPT-5.6 Luna·high · Grok 4.20·think · Grok 4.3·high · Gemini 4 Argon·high · GPT-5.5·high · GPT-6.1 Terra·high · Claude Haiku 5.5·high · Claude Fable 5.2·high · GPT-6 Astra·high · Claude Sonnet 5.5·high |
| pokelike llm-bench harness v4 (badges mean, 50 seeds) pokelike | pokelike llm-bench harness v4 (Pokemon-style game, 50 seeds 10000-10049, 1 pass per model, set notes=12 only); metric badges_mean of 50 runs (0-8 cap); tokens per run from the pass JSON; $/run computed by the README table from published per-token prices at print time; runs 2026-08-21 and 2026-08-22; saved repo pierpierpy/pokelike.xyz.bot; read 2026-10-20 ✓ | xmodel | DeepSeek V4 Flash 0731 build·high · Gemini 3.7 Flash·medium |
| ponytail caveman arm github-ponytail | promptfoo · 5 tasks ×30, caveman skill, cost only ✓ | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 4.8·nothink |
| ponytail skill arm github-ponytail | promptfoo · 5 tasks ×30, ponytail skill, 3 Claude models ✓ | xmodel | Claude Haiku 4.5·nothink · Claude Sonnet 4.6·nothink · Claude Opus 4.8·nothink · GPT-5.5·medium |
| PostTrainBench (GLM-5.2 card, evaluated by PostTrainBench) zhipu | PostTrainBench, 1M context 128K output; board evaluation by posttrainbench.com; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · Gemini 3.1 Pro·high |
| PostTrainBench v1.2 (weighted avg, 6 benchmarks x 4 base models) posttrainbench | PostTrainBench leaderboard results v1.2 (posttrainbench.com/?version=v1.2, changelog 30 Sep / 6 Oct 2026): CLI agent post-trains Qwen3-1.7B, Qwen3-4B, SmolLM3-3B, Gemma-3-4B for 10 h on one H100; score = weighted mean of AIME25, Arena-Hard writing, GPQA Main, GSM8K, HealthBench, HumanEval (BFCL dropped), final evals averaged over up to 5 seeds, contamination judges majority of 3; mean over n runs (2-3); same-benchmark-as:posttrain (paper v1 snapshot); read 2026-10-20 ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 4.8·max · Claude Opus 4.8·high · Claude Opus 4.7·xhigh · GPT-5.5·xhigh · Grok 4.5·high |
| PremiseCheck/ReviewFact (NLP reviews, 6,466 comments), holistic judgment by GPT models arxiv-2610.21046 | ReviewFact NLP subset (arXiv 2610.21046, 15 Oct 2026): each peer-review comment checked on its own by a holistic LLM judgment; score = AUROC % (NLP column of Table 1; ML and natural-science columns are other subsets); cost = Table 5 cost per comment (USD); the authors' own system Jev/PremiseCheck is compared in the paper (not a row); read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.5·low · GPT-5.5·nothink · GPT-5.4 mini·low |
| prinzbench (private legal-research + search benchmark by a single author), leaderboard of 18 Jul 2026 github-prinz-ai-prinzbench | prinzbench (github.com/prinz-ai/prinzbench README 'Last updated on July 18, 2026'): 33 author-written questions (25 legal research + 8 needle search) asked 3 times, pass@1, graded by the author without blinding; score = full score x/99; access through chat apps or terminals at different dates (Jan-Jul 2026); '*' = early access before public release; the author cautions that Pro models use parallel test-time compute; read 2026-10-20 ✓ | xmodel | GPT-5.4·xhigh · Claude Fable 5·max · GPT-5.3-Codex·high · Kimi K3·max · Claude Opus 4.8·max · Gemini 3 Flash·high · Gemini 3 Pro·high |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product given, one-shot baseline github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·low · Claude Sonnet 5.5·low · Claude Haiku 5.5·low |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product given, Wiggum harness github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·low · Claude Sonnet 5.5·low · Claude Haiku 5.5·low |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product hidden, one-shot baseline github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·low · Claude Sonnet 5.5·low · Claude Haiku 5.5·low |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product hidden, Wiggum harness github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·low · Claude Sonnet 5.5·low · Claude Haiku 5.5·low |
| ProgramBench 166 (S55) anthropic | Sonnet 5.5 card p118 · mini-swe-agent, 3 models, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Sonnet 5·max · Claude Opus 5.5·max |
| ProgramDistill-300 arxiv-2609.18805 | arXiv 2609.18805 · 300 web-app repair tasks, R2E-Gym harness, Opus 5 vs Sonnet 5 at high, tokens × LiteLLM list ✓ | xmodel | GPT-6 Astra·high · Claude Opus 5·high · GPT-5.6 Sol·high · Grok 4.6·high · Claude Sonnet 5·high · GPT-5.3-Codex·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Gemini 3.1 Pro·high |
| ProofBench v1.0-era values (vals.ai, Epoch relay) · accuracy % epoch.ai (vals.ai ProofBench v1.0-era) | ProofBench v1.0-era values relayed by Epoch AI (proofbench_external.csv, rows annotated 'v1.0-era value; model absent from the v1.1 board ... Not comparable to v1.1 rows'); vals.ai ProofBench v1.0 methodology (the held group valsproof is the v1.1 re-grade of 2026-08-14); cost per test as relayed; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.4·xhigh · Claude Opus 4.7·max · GPT-5.5·xhigh · Gemini 3.6 Flash·high · GPT-5.4 mini·xhigh · Gemini 3 Flash·high · GPT-5.2·xhigh · Grok 4.20·think · Gemini 3.5 Flash-Lite·high · Grok 4.3·high · GPT-5.1-Codex-Max·high · GPT-5.4 nano·high |
| Protein library (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Sonnet 5·max · Claude Fable 5.1·max · Claude Opus 5.5·max |
| Protein sequence (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Fable 5.1·max · Claude Opus 5.5·max |
| Protein trouble-shoot (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · Gemini 3.1 Pro·high · GPT-6 Astra·max |
| Protein understanding (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · Gemini 3.1 Pro·high · GPT-6 Astra·max |
| Qiita suwa_nobu cache-read qiita-suwanobu | Opus 5 vs Opus 5.5 · same task, warm-cache rounds 2-3, CLI-metered cost ✓ | xmodel | Claude Opus 5·high · Claude Opus 5.5·medium |
| Qiita suwa_nobu cache-write qiita-suwanobu | Opus 5 vs Opus 5.5 · same task, cold-cache round, CLI-metered cost ✓ | xmodel | Claude Opus 5·high · Claude Opus 5.5·medium |
| Qiita Takuya coding set (5 tasks ×3) qiita-takuya | Opus 5.5 high/medium, Opus 5, Fable 5.1; hidden tests 100 % (saturated) ✓ | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| qiita-takuya-reason-hard qiita-takuya | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| qiita-takuya-reason-std qiita-takuya | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| qwen3.7-plus-blog-osworld-verified alibaba | — | xmodel | GPT-5.4·xhigh |
| qwen3.7-plus-blog-screenspot-pro alibaba | — | xmodel | GPT-5.4·xhigh |
| rails-orig-located claude-maintain-rails-evals | — | xmodel | GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · Gemini 4 Argon·high · Claude Sonnet 5.5·low · Claude Opus 5.5·medium |
| rails-v21-located claude-maintain-rails-evals | — | xmodel | GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · Claude Opus 5.5·low |
| rampnet github-rampnet | — | xmodel | Claude Sonnet 5·low · Claude Sonnet 5·high · Claude Opus 5·low · Claude Opus 5·high |
| RampNet Fable github-rampnet | same rig/prompt, effort=low, Anthropic 1P path · Fable 5 vs 5.1 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5.1·low |
| reactbench reactbench | — | xmodel | GPT-5.6 Sol·max · Claude Fable 5.1·max · Claude Fable 5.1·xhigh · Claude Fable 5·max · Claude Fable 5·xhigh · Claude Opus 5·high · Claude Opus 5·max · GPT-5.6 Sol·xhigh · Claude Fable 5.1·high · GPT-5.6 Terra·max · Claude Fable 5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·high · Claude Opus 5·xhigh · Claude Opus 5·medium · DeepSeek V4.1 Flash·max · GPT-5.6 Luna·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · Claude Fable 5·low · GLM-5.3-Flash·max · Claude Opus 4.8·max · Claude Opus 4.8·xhigh · Gemini 3.8 Flash·high · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · Claude Sonnet 5·xhigh · DeepSeek V4 Pro 0813 build·max · GPT-5.6 Terra·medium · Claude Sonnet 5·max · Claude Fable 5.1·medium · Claude Sonnet 5·high · GPT-5.6 Terra·low · GLM-5.3·max · Claude Fable 5.1·low · Claude Opus 4.8·medium · Claude Opus 5·low · Claude Opus 4.8·high · Gemini 3.1 Pro·high · Claude Sonnet 5·medium · Claude Opus 4.8·low · Gemini 3.1 Pro·medium · Kimi K2.7 Code·solo · GPT-5.6 Luna·medium · Claude Sonnet 5·low · Gemini 3.1 Pro·low · GPT-5.6 Luna·low · GLM-5.2·high · Claude Opus 5.5·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · Claude Opus 5.5·low · DeepSeek V4.1 Flash 1017 build·max |
| reactbench-fix reactbench | — | xmodel | GPT-5.6 Sol·max · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·max · GPT-5.6 Terra·xhigh · Claude Fable 5.1·max · Claude Opus 5·high · GPT-5.6 Sol·high · GPT-5.6 Terra·high · Claude Fable 5.1·xhigh · Claude Fable 5·xhigh · Claude Opus 5·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · Gemini 3.8 Flash·high · GPT-5.6 Sol·medium · Claude Opus 5·medium · Claude Opus 5·xhigh · Claude Sonnet 5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·low · Claude Fable 5.1·high · Claude Fable 5.1·low · Claude Fable 5·low · Gemini 3.1 Pro·high · GPT-5.6 Terra·medium · Claude Fable 5.1·medium · Claude Fable 5·high · Claude Fable 5·max · Claude Opus 4.8·medium · Claude Sonnet 5·max · GPT-5.6 Luna·high · Claude Opus 4.8·max · Claude Opus 4.8·xhigh · Claude Opus 5·low · GPT-5.6 Sol·low · Claude Opus 4.8·high · Claude Sonnet 5·high · GLM-5.3·max · Claude Sonnet 5·medium · Gemini 3.1 Pro·medium · GPT-5.6 Luna·medium · GLM-5.3-Flash·max · Kimi K2.7 Code·solo · Gemini 3.1 Pro·low · Claude Opus 4.8·low · Claude Sonnet 5·low · GPT-5.6 Luna·low · GLM-5.2·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·medium · Claude Opus 5.5·max · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·max · Claude Opus 5.5·low |
| reactbench-write reactbench | — | xmodel | Claude Fable 5·max · Claude Fable 5.1·max · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · Claude Fable 5·xhigh · Claude Opus 5·max · Claude Opus 5·high · Claude Fable 5·high · GPT-5.6 Sol·max · GPT-5.6 Terra·max · Claude Opus 5·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · Claude Opus 5·medium · GLM-5.3-Flash·max · GPT-5.6 Sol·low · DeepSeek V4.1 Flash·max · GPT-5.6 Luna·max · Claude Fable 5·low · Claude Opus 4.8·max · Claude Opus 4.8·xhigh · GPT-5.6 Luna·high · GPT-5.6 Sol·medium · Claude Sonnet 5·high · Claude Sonnet 5·max · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·medium · Claude Fable 5.1·medium · Gemini 3.8 Flash·high · GLM-5.3·max · Claude Sonnet 5·xhigh · DeepSeek V4 Pro 0813 build·max · GPT-5.6 Terra·low · Claude Fable 5.1·low · Claude Opus 4.8·high · Claude Opus 4.8·medium · Claude Opus 5·low · Claude Opus 4.8·low · Gemini 3.1 Pro·high · Claude Sonnet 5·medium · Kimi K2.7 Code·solo · Gemini 3.1 Pro·medium · GPT-5.6 Luna·medium · Claude Sonnet 5·low · Gemini 3.1 Pro·low · GPT-5.6 Luna·low · GLM-5.2·high · Claude Opus 5.5·max · Claude Opus 5.5·xhigh · Claude Opus 5.5·high · Claude Opus 5.5·medium · DeepSeek V4.1 Flash 1017 build·low · Claude Opus 5.5·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max |
| Reader-highlight sentence ranking (215 dev pages), zero-shot arxiv-2610.17768 | average-precision difference vs the cached gemini-3.1-pro-preview reference, first sample, one scoring run; low reasoning effort; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·low · Claude Fable 5.1·low · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6.1 Sol·low · Gemini 3.8 Flash·medium |
| RealityBench acceptability (1,378 frames) arxiv-2610.14397 | zero-shot VLM acceptability F1 against the rule scorer; cost = whole set at developer list price, thinking at output rate; read 2026-10-15 ✓ | xmodel | Gemini 3.8 Flash·medium · GPT-6 Sol·medium · GPT-6 Luna·medium · Claude Fable 5.1·high · Claude Opus 5·high · Grok 4.6·high · DeepSeek V4.1 Flash·high |
| Reddit 1x0poki (smith2008): Three Little Pigs SVG film, blind cross-judged reddit-smith2008 | 20-s film of 600 SVG frames, each model in Claude Code, one run; each model scored all three films blind (frame sheets); score /10; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·high · Claude Opus 5.5·high · Claude Haiku 5.5·high |
| Reddit 1x132yy (Escobar747): cross-language agentic task in a real Rust repo reddit-Escobar747 | one agentic task (move logic across a language boundary following repo conventions); cost per task as self-reported (Sonnet: at list prices); qualitative score only; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·medium · Claude Haiku 5.5·high |
| Reddit 1x1jbry (AlexXx52): one read-only Unity investigation, subscription allowance reddit-AlexXx52 | same read-only investigation of a Unity 6 project on each consumer subscription (Claude Pro for Claude); 'cost' = % of the subscription's own usage allowance (NOT USD, not comparable across providers); score = author's subjective quality /10; 1 run; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·medium · GPT-6.1 Sol·medium · Claude Opus 5.5·medium · Claude Opus 4.8·medium · Gemini 3.1 Pro·high · Claude Sonnet 5.5·high · GPT-6.1 Sol·high · Claude Opus 5.5·high · Claude Opus 4.8·high · Claude Sonnet 4.6·high · GPT-5.6 Sol·medium · GPT-6 Astra·medium · GPT-5.6 Sol·high · GPT-6 Astra·high |
| Reddit 1x1nsaz (emarkosov): 5 web tasks (3 research + 2 fact-check) cost reddit-emarkosov | 5 web tasks as subagents, high effort; cost for all 5 tasks as self-reported; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh |
| Reddit 1x1nsaz (emarkosov): code search task, Haiku/Sonnet/Opus 5.5 as subagents reddit-emarkosov | re-run of a real task (find every file for a feature; 48 true files) on each model as subagent, high effort; score = files found of 48; cost as self-reported; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh |
| reddit-1xsuixt-pelican-go-all-out | — | xmodel | Claude Fable 5.2·low · Claude Opus 5.5·medium |
| reddit-1xtq03o-md-rewrite-quality-norm | — | xmodel | Claude Fable 5.2·low · Claude Opus 5.5·medium · Claude Fable 5.2·medium · Claude Opus 5.5·high · Claude Fable 5.2·high · Claude Opus 5.5·xhigh · Claude Fable 5.2·xhigh · Claude Opus 5.5·max |
| ReguSim: financial manipulation detection, overall F1 arxiv-2608.19974 | ReguSim (HKUST, arXiv 2608.19974 v2 12 Oct 2026) Table 5: 900 targets, LLMs receive target-marked windows with neutral account identifiers; overall F1 (%) over 45 cells; supplemental campaign of 29-30 Sep 2026 through one hosted gateway (zctotoken.com); Flash 99.3% valid outputs; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·high · DeepSeek V4.1 Flash·high |
| ReLE 专业能力 (professional ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/专业能力.md (专业能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Gemini 4 Argon·medium · Kimi K2.7 Code·solo · GPT-6.1 Sol·medium · Claude Opus 5·high · Gemini 3.8 Flash·medium · GPT-6 Sol·medium · Grok 4.5·high · GPT-5.4·high · Grok 4.6·high · Claude Sonnet 5.5·high · GPT-6 Luna·medium · Grok 4.7·high · Claude Opus 4.8·nothink · GPT-5.4 mini·high · GPT-5.4 nano·high |
| ReLE 医疗与心理健康 (medical/mental health) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/医疗与心理健康.md (医疗与心理健康); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Kimi K2.7 Code·solo · Gemini 3.8 Flash·medium · Claude Opus 5·high · Gemini 4 Argon·medium · GPT-6 Sol·medium · Grok 4.6·high · GPT-6.1 Sol·medium · GPT-5.4·high · Claude Sonnet 5.5·high · Grok 4.7·high · GPT-6 Luna·medium · Grok 4.5·high · Claude Opus 4.8·nothink · GPT-5.4 mini·high · GPT-5.4 nano·high |
| ReLE 总分 (total) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/total.md (alldata); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Gemini 4 Argon·medium · Claude Opus 5·high · Gemini 3.8 Flash·medium · Claude Sonnet 5.5·high · Kimi K2.7 Code·solo · GPT-5.4·high · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.7·high · Claude Opus 4.8·nothink · Grok 4.5·high · Grok 4.6·high · GPT-6 Luna·medium · GPT-5.4 mini·high · GPT-5.4 nano·high |
| ReLE 推理与数学计算 (reasoning/math) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/推理与数学计算.md (推理与数学计算); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Claude Opus 5·high · Claude Sonnet 5.5·high · Gemini 4 Argon·medium · Gemini 3.8 Flash·medium · GPT-6.1 Sol·medium · Kimi K2.7 Code·solo · GPT-5.4·high · GPT-5.4 nano·high · Grok 4.5·high · Claude Opus 4.8·nothink · Grok 4.6·high · Grok 4.7·high · GPT-5.4 mini·high · GPT-6 Sol·medium · GPT-6 Luna·medium |
| ReLE 教育 (education) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/教育.md (教育); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Gemini 4 Argon·medium · Claude Opus 5·high · Gemini 3.8 Flash·medium · GPT-6.1 Sol·medium · Kimi K2.7 Code·solo · GPT-6 Sol·medium · GPT-5.4·high · Claude Sonnet 5.5·high · Grok 4.5·high · Grok 4.6·high · Claude Opus 4.8·nothink · GPT-6 Luna·medium · Grok 4.7·high · GPT-5.4 mini·high · GPT-5.4 nano·high |
| ReLE 法律与行政公务 (law/admin) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/法律与行政公务.md (法律与行政公务); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · GPT-6.1 Sol·medium · Grok 4.5·high · Claude Opus 5·high · Kimi K2.7 Code·solo · GPT-6 Luna·medium · Grok 4.6·high · Gemini 3.8 Flash·medium · GPT-6 Sol·medium · GPT-5.4·high · Claude Sonnet 5.5·high · Gemini 4 Argon·medium · Grok 4.7·high · Claude Opus 4.8·nothink · GPT-5.4 mini·high · GPT-5.4 nano·high |
| ReLE 语言与指令遵从 (language/IF) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/语言与指令遵从.md (语言与指令遵从); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Gemini 4 Argon·medium · GPT-6.1 Sol·medium · GPT-6 Luna·medium · Gemini 3.8 Flash·medium · Kimi K2.7 Code·solo · GPT-5.4·high · Grok 4.7·high · GPT-6 Sol·medium · Claude Opus 5·high · GPT-5.4 mini·high · Grok 4.6·high · Claude Sonnet 5.5·high · Grok 4.5·high · Claude Opus 4.8·nothink · GPT-5.4 nano·high |
| ReLE 通用能力 (general ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/通用能力.md (通用能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Claude Opus 5·high · Gemini 4 Argon·medium · Gemini 3.8 Flash·medium · Claude Sonnet 5.5·high · Grok 4.7·high · Claude Opus 4.8·nothink · GPT-5.4·high · Kimi K2.7 Code·solo · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.5·high · GPT-6 Luna·medium · Grok 4.6·high · GPT-5.4 mini·high · GPT-5.4 nano·high |
| ReLE 金融 (finance) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/金融.md (金融); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | DeepSeek V4.1 Flash·high · Gemini 4 Argon·medium · Kimi K2.7 Code·solo · GPT-5.4·high · GPT-6.1 Sol·medium · GPT-6 Sol·medium · Grok 4.5·high · Grok 4.6·high · Gemini 3.8 Flash·medium · Claude Opus 5·high · Claude Sonnet 5.5·high · GPT-6 Luna·medium · Grok 4.7·high · Claude Opus 4.8·nothink · GPT-5.4 mini·high · GPT-5.4 nano·high |
| renchris review s55 + o55 rejudge github-renchris-infra | claude -p no-tools, CC 2.1.284 · same 9-brief corpus as renchris-o55 (7 scored / 36 anchored defects), Sonnet 5.5 low→max (new) judged in the same pass as a re-judging of the held Opus 5.5 medium/high/xhigh outputs; 3 blind judges, strict majority; CLI list cost for the Sonnet 5.5 cells only ✓; read 2026-10-01 ✓ | xmodel | Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5·max · Claude Fable 5.1·high |
| renchris synthesis-worker probe (Sonnet 5.5 vs Opus 5.5 xhigh) github-renchris-infra | 6 paired synthesis briefs, 130 key items, 1 cell per arm/brief; judges 2x Opus 5.5 + 1x Opus 5 (claude-judge); mean score /10 over all 18 judge rows; cost/tokens summed over the 6 cells; read 2026-10-01 ✓ | xmodel | Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Opus 5.5·xhigh |
| renkon40 · 50-mail triage note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, 50-mail triage (49 scored); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | xmodel | GPT-6 Luna·high · GPT-6 Sol·high · Claude Opus 5.5·high · Claude Fable 5.1·high · GPT-6 Astra·high |
| renkon40 · sycophancy test, 20 questions note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, sycophancy test, 20 questions; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | xmodel | Claude Opus 5.5·high · Claude Fable 5.1·high · Claude Opus 5·high · Claude Fable 5·high · Gemini 3.8 Flash·high · GPT-6 Astra·high · GPT-6 Sol·high · GPT-6 Luna·high · Claude Sonnet 5.5·high |
| ReplaySCM Block-order (100 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Block-order (100 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · Gemini 3.1 Pro·medium · Grok 4.20·think · Kimi K2 Thinking·solo |
| ReplaySCM Hidden-order Hid-Full (250 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Hidden-order Hid-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · Gemini 3.1 Pro·medium · Grok 4.20·think · Kimi K2 Thinking·solo |
| ReplaySCM Ord-Full (250 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Ord-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · Gemini 3.1 Pro·medium · Grok 4.20·think · Kimi K2 Thinking·solo |
| Retort brazil-soccer-mcp (clojure): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (cpp): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (csharp): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (elixir): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (erlang): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (go): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (java): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (objc): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (python): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (rust): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (swift): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort brazil-soccer-mcp (typescript): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (c): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (clojure): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (cpp): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (elixir): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (erlang): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (java): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (objc): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (rust): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (swift): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| Retort rest-api-crud (typescript): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Claude Opus 5.5·low · Claude Haiku 5.5·low |
| roblox-leetcode-easy hf-roblox-llm-leaderboard | — | xmodel | Claude Fable 5·medium · Claude Fable 5.1·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium · Claude Sonnet 5.5·medium · DeepSeek V4 Pro 0813 build·high · Gemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · GPT-5.4 nano·high · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.5·medium · Grok 4.6·medium · GLM-5.2·high · GLM-5.3·high · GLM-5.3-Flash·high |
| roblox-leetcode-hard hf-roblox-llm-leaderboard | — | xmodel | Claude Fable 5·medium · Claude Fable 5.1·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium · Claude Sonnet 5.5·medium · DeepSeek V4 Pro 0813 build·high · Gemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · GPT-5.4 nano·high · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.5·medium · Grok 4.6·medium · GLM-5.2·high · GLM-5.3·high · GLM-5.3-Flash·high |
| roblox-leetcode-medium hf-roblox-llm-leaderboard | — | xmodel | Claude Fable 5·medium · Claude Fable 5.1·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium · Claude Sonnet 5.5·medium · DeepSeek V4 Pro 0813 build·high · Gemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · GPT-5.4 nano·high · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.5·medium · Grok 4.6·medium · GLM-5.2·high · GLM-5.3·high · GLM-5.3-Flash·high |
| roblox-qa-freeform hf-roblox-llm-leaderboard | — | xmodel | Claude Fable 5·medium · Claude Fable 5.1·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium · Claude Sonnet 5.5·medium · DeepSeek V4 Pro 0813 build·high · Gemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · GPT-5.4 nano·high · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.5·medium · Grok 4.6·medium · GLM-5.2·high · GLM-5.3·high · GLM-5.3-Flash·high |
| roblox-qa-multiple-choice hf-roblox-llm-leaderboard | — | xmodel | Claude Fable 5·medium · Claude Fable 5.1·medium · Claude Opus 4.8·medium · Claude Opus 5·medium · Claude Opus 5.5·medium · Claude Sonnet 5·medium · Claude Sonnet 5.5·medium · DeepSeek V4 Pro 0813 build·high · Gemini 3.1 Pro·medium · Gemini 3.7 Flash·medium · Gemini 3.8 Flash·high · GPT-5.4 nano·high · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·medium · GPT-5.6 Terra·medium · GPT-6 Astra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·medium · Grok 4.5·medium · Grok 4.6·medium · GLM-5.2·high · GLM-5.3·high · GLM-5.3-Flash·high |
| robocolis-agent2act arxiv-2610.27865 | — | xmodel | Claude Opus 5.5·high · GPT-6 Astra·high · GPT-6.1 Sol·high · Gemini 4 Argon·medium |
| Roboflow Playground Vision Evals, Counting (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Counting; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-counting) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Qwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink |
| Roboflow Playground Vision Evals, Identification (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Identification; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-identification) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Qwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink |
| Roboflow Playground Vision Evals, Object Detection (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Object Detection; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-detection) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Qwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink |
| Roboflow Playground Vision Evals, OCR (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: OCR; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-ocr) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Gemini 4 Argon·low · GPT-6 Astra·low · Claude Opus 5.5·low · GPT-6.1 Sol·low · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Claude Sonnet 5.5·low · Gemini 3.6 Flash·low · GPT-6 Sol·low · Gemini 3.1 Pro·low · Claude Fable 5.1·low · GPT-5.6 Sol·low · Claude Fable 5·low · Claude Opus 5·low · GPT-6 Luna·low · Claude Haiku 5.5·low · GPT-5.5·low · GPT-5.6 Terra·low · Gemini 3 Flash·low · GPT-5.6 Luna·low · Grok 4.7·low · Gemini 3.5 Flash-Lite·low · Qwen3.8 Flash·nothink · Grok 4.6·low · Claude Opus 4.8·low · Grok 4.5·low · Claude Sonnet 5·low · GPT-5.4 mini·low · Kimi K2.6·nothink · Qwen3.7 Flash·nothink |
| Roboflow Playground Vision Evals, overall (mean of tasks) with cost per sample roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; Overall = page's mean over the tasks scored (4/5 where OCR is missing); cost = 'Avg cost / sample' in USD from price x tokens at the pricing date 2026-10-19 ('~' = estimated from self-host/OpenRouter pricing); read 2026-10-20 ✓ | xmodel | Gemini 4 Argon·low · GPT-6 Astra·low · Claude Opus 5.5·low · GPT-6.1 Sol·low · Gemini 3.7 Flash·low · Gemini 3.8 Flash·low · Claude Sonnet 5.5·low · Gemini 3.6 Flash·low · GPT-6 Sol·low · Gemini 3.1 Pro·low · Claude Fable 5.1·low · GPT-5.6 Sol·low · Claude Fable 5·low · Claude Opus 5·low · GPT-6 Luna·low · Claude Haiku 5.5·low · GPT-5.5·low · GPT-5.6 Terra·low · Gemini 3 Flash·low · GPT-5.6 Luna·low · Grok 4.7·low · Gemini 3.5 Flash-Lite·low · Qwen3.8 Flash·nothink · Grok 4.6·low · Claude Opus 4.8·low · Grok 4.5·low · Claude Sonnet 5·low · GPT-5.4 mini·low · Kimi K2.6·nothink · Qwen3.7 Flash·nothink |
| Roboflow Playground Vision Evals, Reasoning (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: Reasoning; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Qwen3.6-27B·nothink · Qwen3.5-27B·nothink · Qwen3.5-35B-A3B·nothink |
| RoboQuest (10 tasks, 50 episodes per cell), Inspect Robots harness arxiv-2610.10388 | 200-decision budget per episode, success rate over 50 episodes x 10 tasks; cost per episode at list prices; read 2026-10-15 ✓ | xmodel | GPT-6 Astra·medium · Claude Opus 5.5·medium · GPT-6.1 Sol·medium · Claude Fable 5.1·medium · Gemini 3.8 Flash·high |
| RoTA-Bench (interactive 3D projection alignment, 100 tasks) arxiv-2610.17665 | RoTA-Bench (arXiv 2610.17665, 13 Oct 2026): 100 tasks, interactive 3D projection alignment, 48 decisions max, same prompt/harness for every model; score = success % (IoU >= 0.9; Final IoU and steps are other columns); cost = Table 20 total USD over the 100 tasks estimated by the authors from recorded tokens and published list prices; Temperature 0, reasoning effort medium where supported; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·medium · GPT-6 Astra·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · Claude Opus 5·medium |
| rotabench-final-iou arxiv-2610.17665 | — | xmodel | GPT-5.6 Sol·medium · GPT-6 Astra·medium · GPT-5.6 Terra·medium · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · Claude Opus 5·medium |
| RSI Index (OpenAI GPT-5.6 post chart) openai | RSI Index, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| rt-v1.1-audio-proofread heydonna | — | xmodel | Gemini 3.8 Flash·low · Gemini 4 Argon·low · Gemini 3.1 Flash-Lite·minimal |
| rt-v1.1-text-sc heydonna | — | xmodel | Claude Sonnet 5.5·high · GPT-6.1 Sol·medium |
| rtk-base arxiv-2607.12161 | — | xmodel | Claude Sonnet 5·low · Claude Sonnet 5·high · Claude Opus 4.8·low · Claude Opus 4.8·high |
| rtx4090-local-vs-cloud: 8 manufacturing-domain tasks, programmatic grading, cloud Claude rows via Claude Code github-rtx4090-local-vs-cloud | Howard159357/rtx4090-local-vs-cloud-coding-benchmark README (15 Oct 2026); 3 runs per setup, Claude Code CLI with built-in shell/web/plugins disabled; subscription (no API cost); score = mean of 3 runs as printed; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium · Claude Haiku 5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Luna·xhigh · GPT-6 Luna·medium |
| rubench-claude-code arxiv-2607.06411 | — | xmodel | Claude Opus 4.8·xhigh · Claude Sonnet 5·xhigh |
| Rubric transfer: expert score, ordinary rubric arxiv-2610.10809 | Rubric-transfer study on HealthBench-style typical split (n=105, 15 per theme over 7 themes): responses from a panel of response models are scored by judge Gemini 3.8 Flash (reasoning effort medium, majority of 3) against the expert rubric ('ordinary') or a foreign rubric; expert score 0..1 with 95% paired bootstrap CI; sampling temperature = each provider's default; ordinary expert rubric; read 2026-10-19 ✓ | xmodel | GLM-5.3-Flash·low · Grok 4.20·nothink |
| rubric-expert-share-lost arxiv-2610.10809 | — | xmodel | GLM-5.3-Flash·low · Grok 4.20·nothink |
| RuneBench github-runebench | Harbor+Modal · 16 skills, 30-min budget, XP metric; mispriced Opus 5 cells omitted ✓ | xmodel | Claude Fable 5·xhigh · Claude Fable 5.1·xhigh · Claude Fable 5·high · Claude Fable 5.1·high · Claude Opus 4.7·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·max · Claude Sonnet 5·xhigh · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·xhigh · Claude Haiku 5.5·high · GPT-6 Astra·medium · GPT-5.6 Terra·high · GPT-5.4 mini·high · GPT-5.6 Sol·high · GPT-5.6 Luna·high · GPT-5.5·xhigh · GPT-5.6 Terra·xhigh · GPT-6 Sol·high · GPT-5.4·high · GPT-5.6 Sol·xhigh · GPT-6 Astra·high · GPT-6 Luna·medium · GPT-5.6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Sol·medium |
| s7t-bsky-ironcurtain-fps-gun bluesky | — | xmodel | Claude Fable 5.2·xhigh · Claude Opus 5.5·xhigh |
| s7t-claudebattle-auric-final github-eval-forge | — | xmodel | Claude Fable 5.2·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high |
| s7t-claudebattle-auric-quality github-eval-forge | — | xmodel | Claude Fable 5.2·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Haiku 5.5·high |
| s7t-claudebattle-auric-truthness github-eval-forge | — | xmodel | Claude Fable 5.2·high · Claude Opus 5.5·high · Claude Haiku 5.5·high |
| s7t-nbl-mnist-step1 notesbylex | — | xmodel | Claude Fable 5.2·high · Claude Fable 5.2·low · Claude Fable 5.1·high · Claude Opus 5.5·high |
| s7t-nbl-mnist-step15 notesbylex | — | xmodel | Claude Fable 5.2·high · Claude Fable 5.2·low · Claude Fable 5.1·high · Claude Opus 5.5·high |
| s7t-techkwon-motion github-techkwon | — | xmodel | Claude Opus 5.5·high · GPT-6.1 Sol·high |
| s7t-techkwon-physics github-techkwon | — | xmodel | Claude Opus 5.5·high · GPT-6.1 Sol·high |
| s7t-techkwon-svg github-techkwon | — | xmodel | Claude Opus 5.5·high · GPT-6.1 Sol·high |
| s7t-techkwon-three github-techkwon | — | xmodel | Claude Opus 5.5·high · GPT-6.1 Sol·high |
| s7t-wsp-exp5-seaquest huggingface-shawnghu | — | xmodel | Claude Fable 5.2·high · Claude Fable 5.2·xhigh |
| sakana-rhi47 arxiv-2607.15524 | — | xmodel | Claude Opus 4.7·high · Claude Opus 4.7·max · Claude Opus 4.7·xhigh |
| sakana-rhi48 arxiv-2607.15524 | — | xmodel | Claude Opus 4.8·high · Claude Opus 4.8·max · Claude Opus 4.8·xhigh |
| Scale EnigmaEval (Wayback 2026-10-16, current 5-model board) scale-seal | https://labs.scale.com/leaderboard/enigma_eval; Wayback capture 20261016182047 (saved gpt-5.1-chat-latest/src/scale/dec_enigma_eval.html); score = board score % with 95% CI; puzzle hunts; current board (models listed with a -high / -xhigh suffix); read 2026-10-20 ✓ | xmodel | Claude Fable 5·high · GPT-5.6 Sol·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Claude Opus 4.8·xhigh |
| Scale Humanity's Last Exam text-only (Wayback 2026-10-07) scale-seal | https://labs.scale.com/leaderboard/humanitys_last_exam_text_only; Wayback capture 20261007135520 (saved gpt-5.1-chat-latest/src/scale/dec_humanitys_last_exam_text_only.html); score = board score % with 95% CI; HLE text-only, board page of 2026-10-07 (HLE-Rolling note of 17 Sep 2026); read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high · Claude Fable 5.1·xhigh · GPT-5.4·xhigh |
| SCBench scbench.ai | SCBench, agentic coding through successive checkpoints in Claude Code · strict solve rate in %, cost per checkpoint (USD); effort from the model label (High); the Claude Code version differs per row; read 2026-09-30 ✓ | xmodel | GPT-5.3-Codex·high · GPT-5.6 Sol·xhigh · GPT-6 Astra·xhigh · Claude Opus 5.5·medium · GPT-6 Astra·medium · GPT-6.1 Sol·medium · GPT-6 Sol·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · GPT-6 Sol·medium |
| scbench-iso scbench.ai | — | xmodel | GPT-5.6 Sol·xhigh · GPT-6 Astra·xhigh · Claude Opus 5.5·medium · GPT-6 Astra·medium · GPT-6.1 Sol·medium · GPT-6 Sol·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · GPT-6 Sol·medium |
| SCDBench smart-contract decompilation, with one zero-shot compilation-repair pass (dagger rows) arxiv-2605.29059 | SCDBench (arXiv 2605.29059, 27 May 2026): 600 real Solidity contracts (200 easy/medium/hard), with one zero-shot compilation-repair pass (dagger rows); score = semantic consistency (% of 14,553 public functions passing replayable semantic checkpoints; Table 4; the earlier stages format/compile/ABI are other cumulative columns); cost = Table 9 average API cost per contract at OpenRouter prices at the time; read 2026-10-20 ✓ | xmodel | Claude Opus 4.7·high · GPT-5.3-Codex·high |
| SCDBench smart-contract decompilation, zero-shot decompilation arxiv-2605.29059 | SCDBench (arXiv 2605.29059, 27 May 2026): 600 real Solidity contracts (200 easy/medium/hard), zero-shot decompilation; score = semantic consistency (% of 14,553 public functions passing replayable semantic checkpoints; Table 4; the earlier stages format/compile/ABI are other cumulative columns); cost = Table 9 average API cost per contract at OpenRouter prices at the time; read 2026-10-20 ✓ | xmodel | Claude Opus 4.7·high · GPT-5.3-Codex·high |
| scf52-arc3p anthropic | — | xmodel | Claude Fable 5.2·high · Claude Fable 5.2·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6 Astra·high · GPT-6.1 Sol·high |
| scf52-arc3s anthropic | — | xmodel | Claude Fable 5.2·high · Claude Fable 5.2·max |
| scf52-autom anthropic | — | xmodel | Gemini 4 Carbon·high · Claude Fable 5.2·low · Claude Fable 5.2·max · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Opus 5.5·max |
| scf52-bcadn anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| scf52-bcadt anthropic | — | xmodel | Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Fable 5.2·max |
| scf52-briefe anthropic | — | xmodel | Gemini 4 Carbon·high · Claude Fable 5.2·max · Claude Fable 5.2·xhigh |
| scf52-chartn anthropic | — | xmodel | Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Gemini 4 Carbon·high · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max |
| scf52-charto anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Gemini 4 Argon·high · GPT-6 Astra·max |
| scf52-chartr anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Gemini 4 Argon·high · GPT-6 Astra·max |
| scf52-chartt anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| scf52-cve anthropic | — | xmodel | Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max |
| scf52-draco anthropic | — | xmodel | Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max |
| scf52-fc20 anthropic | — | xmodel | Claude Fable 5.2·max · Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh |
| scf52-fswe2 anthropic | — | xmodel | Gemini 4 Carbon·high · Claude Fable 5.2·max · GPT-6 Astra·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · Gemini 4 Argon·high |
| scf52-gdpval anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Fable 5.2·xhigh |
| scf52-gmmlu anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max |
| scf52-hb anthropic | — | xmodel | Claude Fable 5.2·max · Claude Fable 5.2·low · Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Haiku 5.5·max |
| scf52-hba anthropic | — | xmodel | Claude Fable 5.2·max · Claude Fable 5.2·low · Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Haiku 5.5·max |
| scf52-hbp anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · GPT-6 Astra·max · Claude Fable 5.1·max · Claude Haiku 5.5·max |
| scf52-hbpa anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · GPT-6 Astra·max · Claude Fable 5.1·max · Claude Haiku 5.5·max |
| scf52-hlen anthropic | — | xmodel | Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max |
| scf52-hlet anthropic | — | xmodel | GPT-6 Astra·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max |
| scf52-milu anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max |
| scf52-offqa anthropic | — | xmodel | Claude Fable 5.2·max · Claude Fable 5.1·max · Claude Opus 5.5·max |
| scf52-offqap anthropic | — | xmodel | Claude Fable 5.2·max · Claude Fable 5.1·max · Claude Opus 5.5·max |
| scf52-oswp anthropic | — | xmodel | Gemini 4 Carbon·high · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max · GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| scf52-osws anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max · GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| scf52-phys anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · GPT-6 Astra·max · GPT-6 Astra·high · Claude Fable 5.2·low · Claude Fable 5.2·high · Claude Fable 5.2·xhigh |
| scf52-rli anthropic | — | xmodel | Claude Fable 5.2·high · Claude Opus 5.5·high · Claude Fable 5.1·high |
| scf52-robodojo anthropic | — | xmodel | Gemini 4 Argon·high · GPT-6 Astra·max · Claude Fable 5.1·max · Claude Opus 5.5·max · Claude Fable 5.2·max |
| scf52-stemprof anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max |
| scf52-swepro-sub anthropic | — | xmodel | Claude Opus 5.5·high · Claude Opus 5.5·low |
| scf52-tb30 anthropic | — | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·high |
| scf52-tb4 anthropic | — | xmodel | Claude Fable 5.2·max · Claude Opus 5.5·xhigh · Claude Fable 5.1·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GPT-6 Astra·high · GPT-6 Astra·max |
| scf52-tbsci anthropic | — | xmodel | Gemini 4 Carbon·high · Claude Fable 5.2·max · Claude Opus 5.5·max · Claude Fable 5.1·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GPT-6 Sol·max |
| scf52-wandr anthropic | — | xmodel | Claude Fable 5.2·low · Claude Fable 5.2·medium · Claude Fable 5.2·high · Claude Fable 5.2·xhigh · Claude Fable 5.2·max |
| SCHEME (arXiv 2605.29178) · main-task pass rate, side task removed (benign baseline) arxiv-2605.29178 | SCHEME multi-agent coding benchmark, baseline runs with include_side_task=false: 17 instances x 5 epochs = 85 trajectories per model; temperature 1.0; 32,000 max output tokens; rates printed as rounded integers in the text; read 2026-10-20 ✓ | xmodel | Gemini 3.1 Pro·high |
| scheme-sabotage-success arxiv-2605.29178 | — | xmodel | Gemini 3.1 Pro·high |
| schletools anthropic | — | xmodel | Claude Sonnet 5·max · Claude Opus 4.8·max · Claude Sonnet 4.6·low · Claude Sonnet 4.6·high |
| scorch-bench: artillery duel game vs 7 bots, Claude Code subagents (reasoning track), seed 17 github-scorch-bench | smerwin/scorch-bench bench/RESULTS-claude.md (2026-10-09); 8 matches per model on one seed; one fresh Claude Code subagent per round via relay provider; score = win rate %; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| ScreenSpot-Pro (OpenAI GPT-6 Astra post chart) openai | ScreenSpot-Pro; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| Search agent with gold passages kept, answer EM on HOTPOTQA (vanilla retrieval tool) arxiv-2610.05348 | 300 HotpotQA questions, BM25 over wiki-18, vanilla tool; Table 1/4 of arXiv 2610.05348; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·high |
| Search agent with gold passages kept, answer EM on NQ (vanilla retrieval tool) arxiv-2610.05348 | 257 NQ questions, BM25 over wiki-18, vanilla tool; Table 1/4 of arXiv 2610.05348; read 2026-10-15 ✓ | xmodel | Claude Sonnet 5.5·high |
| SEC-Bench Pro (OpenAI GPT-5.6 post chart) openai | SEC-Bench Pro pass@1, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max |
| Senko Flysim senko | Senko Rašić · one-shot flight-sim game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓ | xmodel | GPT-6 Sol·xhigh · Claude Opus 5.5·xhigh · Claude Opus 5·xhigh · GPT-6.1 Terra·xhigh · Claude Haiku 5.5·xhigh |
| Senko RTS senko | Senko Rašić · one-shot RTS game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓ | xmodel | GPT-6 Sol·xhigh · Claude Opus 5.5·xhigh · Claude Opus 5·xhigh · GPT-6.1 Terra·xhigh · Claude Haiku 5.5·xhigh |
| Senko Voxel senko | Senko Rašić · one-shot Minecraft-like game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓ | xmodel | GPT-6 Sol·xhigh · Claude Opus 5.5·xhigh · Claude Opus 5·xhigh · GPT-6.1 Terra·xhigh · Claude Haiku 5.5·xhigh |
| SenseBench (WSD) · prompt p001 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Claude Opus 5·max · Claude Opus 4.8·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·nothink · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.7·medium · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·xhigh · Claude Sonnet 5·high · Claude Sonnet 4.6·low · Claude Haiku 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Haiku 5.5·nothink · Claude Haiku 5.5·xhigh · Claude Opus 5.5·low · Claude Opus 5.5·xhigh · Claude Sonnet 5.5·low · Claude Sonnet 5.5·xhigh · GPT-6 Astra·xhigh · GPT-5.5·xhigh · Gemini 3.8 Flash·low · GPT-5.5·medium · GPT-5.5·high · Gemini 3.8 Flash·high · GPT-6 Sol·low · GPT-6 Sol·xhigh · GPT-5.5·low · GPT-5.6 Sol·xhigh · Gemini 3.1 Pro·high · Gemini 3.1 Pro·low · GPT-5.6 Sol·medium · Gemini 3.1 Pro·medium · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·high · Grok 4.5·high · GPT-5.6 Terra·medium · GPT-5.6 Terra·low · Kimi K2.7 Code·solo · Grok 4.3·medium · Grok 4.3·low · GPT-6 Luna·xhigh · Grok 4.3·high · DeepSeek V4 Pro preview·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·high · GPT-5.6 Terra·nothink · DeepSeek V4 Flash preview·high · GPT-6 Luna·low · Gemini 3.1 Flash-Lite·minimal · GPT-5.6 Luna·medium · GPT-5.4 mini·low · GPT-5.6 Luna·low · GPT-5.6 Luna·nothink · GPT-5.4 nano·low · Gemini 4 Argon·high · Gemini 4 Argon·low · DeepSeek V4.1 Flash·max · Qwen3.8 Max 0902 build·xhigh · GLM-5.3·max · DeepSeek V4.1 Flash·low · Qwen3.8 Max 0902 build·low · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·medium · Qwen3.8-27B·low · GLM-5.3-Flash·low · GPT-6 Luna·nothink · GLM-5.3·low |
| SenseBench (WSD) · prompt p002 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p002 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Claude Opus 4.8·high · Claude Sonnet 4.6·low · Claude Haiku 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Haiku 5.5·nothink · Claude Haiku 5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · Gemini 3 Flash·high · DeepSeek V4 Pro preview·high · DeepSeek V4 Flash preview·high · GPT-5.4 mini·low · Gemini 3.1 Flash-Lite·minimal · GPT-5.4 nano·low · GPT-6 Luna·max · GPT-6 Luna·medium · GPT-6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Luna·low · GPT-6 Luna·nothink |
| SenseBench (WSD) · prompt p003 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p003 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Claude Opus 4.7·medium · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Haiku 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Haiku 5.5·nothink · Claude Haiku 5.5·xhigh · GPT-5.5·high · GPT-5.5·low · GPT-5.5·medium · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.5·nothink · GPT-5.6 Terra·medium · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·high · GPT-5.6 Terra·nothink · GPT-5.4 mini·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·low · GPT-5.6 Luna·nothink · GPT-5.4 nano·low · GPT-6 Luna·max · GPT-6 Luna·high · GPT-6 Luna·medium · GPT-6 Luna·xhigh · GPT-6 Luna·low · GPT-6 Luna·nothink |
| SenseBench (WSD) · prompt p004 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p004 (new prompt variant) · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-10-15 ✓ | xmodel | Claude Haiku 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·max · Claude Haiku 5.5·medium · Claude Haiku 5.5·nothink · Claude Haiku 5.5·xhigh · Claude Opus 4.8·xhigh · GPT-5.5·medium · GPT-5.5·xhigh · GPT-6 Luna·max · GPT-6 Luna·xhigh · GPT-6 Luna·high · GPT-6 Luna·medium · GPT-6 Luna·low · GPT-6 Luna·nothink |
| ServeLearnBench Avg. Hidden (Blind) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Blind; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·high |
| ServeLearnBench Avg. Hidden (CH) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness CH; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·high |
| ServeLearnBench Avg. Hidden (Mem0) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Mem0; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·high |
| ServeLearnBench Avg. Hidden (Oracle) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Oracle; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·high |
| ServeLearnBench Avg. Hidden (RAG) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness RAG; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·high |
| ServeLearnBench Avg. Hidden (SkillOpt) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness SkillOpt; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·high |
| SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies): MCC on primary studies sesr-eval-2610.10633 | SESR-Eval screening of SE systematic reviews (SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies)): LLM title/abstract screening via OpenRouter + Pydantic AI, MCC pooled over primary studies (Accuracy/Precision/Recall/F1/lost evidence and the secondary-study table are other summaries of the same runs: rule 8); cost = USD per secondary study (5% sample) from the secondary-study table; Claude 3.7 Sonnet is an old model whose cost is a previously recorded estimate; arXiv 2610.10633 (Oct 2026); read 2026-10-20 ✓ | xmodel | Claude Opus 4.7·low · Gemini 3.1 Pro·low · GPT-5.5·low · Grok 4.20·think |
| SimpleBench multiple-choice (AVG@5) with cost per run simplebench | SimpleBench >200 private multiple-choice questions (spatio-temporal, social intelligence, trick questions); score = Score (AVG@5) as printed; cost = modelMeta.cpr 'pre-computed cost to run the whole benchmark once (measured tokens x OpenRouter price)' USD; tout = measured output tokens (unit unstated); live board, rows added between 2025-10 and 2026-10-01 (dateAdded printed); read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·xhigh · Kimi K3·max · Kimi K2.7 Code·solo · GPT-5.2 Pro·xhigh · GPT-5.1·high · GPT-5.6 Terra·xhigh · GPT-5.6 Luna·xhigh · GPT-5.2·high · Kimi K2 Thinking·solo |
| Single-cell (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max |
| singularitygate-v2-1-all-80 singularitygate | — | xmodel | Claude Fable 5.2·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · Claude Sonnet 5.5·xhigh · Claude Haiku 5.5·xhigh |
| singularitygate-v2-1-hard-12 singularitygate | — | xmodel | Claude Fable 5.2·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · Claude Sonnet 5.5·xhigh · Claude Haiku 5.5·xhigh |
| SkillHorizon long-horizon business-agent tasks: task success (overall, all reference-workload bands) skillhorizon-2610.14966 | SkillHorizon Table 1 TS Overall: 7 executable MCP business environments, 5 reference-workload bands 64K-1M tokens, 210 executions per model and band (1,050 per model), Harbor + Inspect, context compaction at 200K, temperature/top-p unspecified; TS = proportion of tasks satisfying all final requirements; LGC (local-goal completion) and per-band columns = same runs (rule 8); arXiv 2610.14966 (Oct 2026); read 2026-10-20 ✓ | xmodel | Grok 4.6·medium · Gemini 3.8 Flash·medium · GPT-5.6 Sol·medium |
| slopcode-iso arxiv-2603.24755 | — | xmodel | GPT-5.3-Codex·high · GPT-5.4 mini·high · GPT-5.5·high |
| SOC T1a concentration estimation (MAE g C/kg, lower better) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Terra·nothink · Gemini 3.8 Flash·low · Grok 4.6·low |
| SOC T1b concentration estimation with extra predictors (MAE g C/kg, lower better) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Terra·nothink · Gemini 3.8 Flash·low · Grok 4.6·low |
| SOC T2 stock calculation accuracy (%) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Terra·nothink · Gemini 3.8 Flash·low · Grok 4.6·low |
| SOC T3 pairwise paddock choice accuracy (%) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·nothink · GPT-5.6 Terra·nothink · Gemini 3.8 Flash·low · Grok 4.6·low |
| Spatial transcriptomics (S55) anthropic | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max |
| spatialscout-micro-all arxiv-2610.26659 | — | xmodel | Claude Opus 5.5·xhigh · Claude Sonnet 5.5·xhigh · GPT-6 Astra·xhigh · GPT-6 Astra·medium · GPT-6 Sol·xhigh |
| SpireBench: Slay the Spire Ascension 20 Ironclad, mean final floor reached arxiv-2610.12596 | 10 runs per model on the same held-out seeds, themed reskin with note-taking; Table 1 and Table 26 of arXiv 2610.12596; read 2026-10-15 ✓ | xmodel | Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh · Claude Fable 5.1·low · Claude Fable 5.1·max · Claude Opus 5·low · Claude Opus 5·max · Claude Sonnet 5·low · Claude Sonnet 5·max · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · GPT-6 Astra·low · GPT-6 Astra·max · GPT-5.6 Sol·low · GPT-5.6 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GLM-5.3·low · GLM-5.3·max · GLM-5.3-Flash·low · GLM-5.3-Flash·max · DeepSeek V4.1 Flash·max · Grok 4.6·xhigh |
| StagedWorkspace APEX-Agents file-editing subset, diffs hidden stagedws-2608.18050 | APEX-Agents 57 file-editing tasks, SW-Agent dual views, review-axis arm: workspace diff tools diffs hidden; metric mean rubric score %; arXiv 2608.18050 Table 3c (Aug 2026); read 2026-10-20 ✓ | xmodel | Gemini 3 Flash·high · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh |
| StagedWorkspace APEX-Agents file-editing subset, diffs visible stagedws-2608.18050 | APEX-Agents 57 file-editing tasks, SW-Agent dual views, review-axis arm: workspace diff tools diffs visible; metric mean rubric score %; arXiv 2608.18050 Table 3c (Aug 2026); read 2026-10-20 ✓ | xmodel | Gemini 3 Flash·high · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh |
| StagedWorkspace APEX-Agents mean rubric score, artifact-only (native files, no parsed search) stagedws-2608.18050 | APEX-Agents 452 tasks (480 minus 28 Investment Banking tasks needing EDGAR), SW-Agent harness, arm artifact-only (native files, no parsed search), 3 reruns per task, grader gemini-3-flash-preview at low thinking; metric mean rubric score % (Pass@1 of Table 2b = same runs); cost = mean agent-run USD per task at public list rates for model and run date, parsing/sandbox excluded (Table A.5); arXiv 2608.18050 Tables 2b/3b/A.5 (Aug 2026); read 2026-10-20 ✓ | xmodel | Gemini 3 Flash·high · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh |
| StagedWorkspace APEX-Agents mean rubric score, dual views (native + parsed) stagedws-2608.18050 | APEX-Agents 452 tasks (480 minus 28 Investment Banking tasks needing EDGAR), SW-Agent harness, arm dual views (native + parsed), 3 reruns per task, grader gemini-3-flash-preview at low thinking; metric mean rubric score % (Pass@1 of Table 2b = same runs); cost = mean agent-run USD per task at public list rates for model and run date, parsing/sandbox excluded (Table A.5); arXiv 2608.18050 Tables 2b/3b/A.5 (Aug 2026); read 2026-10-20 ✓ | xmodel | Gemini 3 Flash·high · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh |
| StagedWorkspace APEX-Agents mean rubric score, parsed-only (hash-synced parsed cache, no native files) stagedws-2608.18050 | APEX-Agents 452 tasks (480 minus 28 Investment Banking tasks needing EDGAR), SW-Agent harness, arm parsed-only (hash-synced parsed cache, no native files), 3 reruns per task, grader gemini-3-flash-preview at low thinking; metric mean rubric score % (Pass@1 of Table 2b = same runs); cost = mean agent-run USD per task at public list rates for model and run date, parsing/sandbox excluded (Table A.5); arXiv 2608.18050 Tables 2b/3b/A.5 (Aug 2026); read 2026-10-20 ✓ | xmodel | Gemini 3 Flash·high · GPT-5.4 mini·xhigh · GPT-5.4 nano·xhigh |
| StateM harness on Terminal-Bench 2.1 arxiv-2608.15089 | StateM (arXiv 2608.15089 v1 15 Aug 2026): Terminal-Bench 2.1, 89 tasks x 5 trials = 445 trials, Codex-based agent with the authors' StateM runtime/runbook (GPT profile developed on GPT-5.5, frozen for GPT-5.6; DeepSeek profile adapted with $37.02 of API spend); score = trial success %; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.6 Sol·xhigh |
| statem-tb21-references arxiv-2608.15089 | — | xmodel | GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| stet-board24-oct stet.sh | — | xmodel | GPT-6.1 Sol·medium · Claude Opus 5·medium · Grok 4.7·high · Claude Haiku 5.5·max · GPT-6 Sol·medium · GPT-6 Luna·max · DeepSeek V4.1 Flash·max · Claude Haiku 5.5·low · Claude Opus 4.8·medium · GPT-6 Luna·low |
| stet-pr24 stet.sh | — | xmodel | Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max |
| Supabase evals (with skills) supabase.com | Supabase evals in Claude Code, with skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5·high · GPT-6 Luna·medium · GPT-6 Sol·medium · Grok 4.7·high |
| Supabase evals (without skills) supabase.com | Supabase evals in Claude Code, without skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓ | xmodel | Claude Opus 5.5·high · Claude Sonnet 5·high · GPT-6 Luna·medium · GPT-6 Sol·medium · Grok 4.7·high |
| SuperCLUE 2025年11月 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | GPT-5.2·high · GPT-5.1·high · GLM-4.7·think · Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | GPT-5.2·high · GPT-5.1·high · GLM-4.7·think · Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | GPT-5.2·high · GPT-5.1·high · GLM-4.7·think · Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | GPT-5.2·high · GPT-5.1·high · GLM-4.7·think · Grok 4.1 Fast·think |
| SuperCLUE 2025年11月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年11月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | GPT-5.2·high · GPT-5.1·high · GLM-4.7·think · Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·high · GLM-5·think · GLM-4.7·think · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·high · GLM-5·think · GLM-4.7·think · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·high · GLM-5·think · GLM-4.7·think · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·high · GLM-5·think · GLM-4.7·think · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·high · GLM-5·think · GLM-4.7·think · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| SuperCLUE 2025年度测评 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.2·high · GLM-5·think · GLM-4.7·think · Kimi K2 Thinking·solo · Grok 4.1 Fast·think |
| SuperCLUE 2026年3月 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3 Flash·high · Grok 4.20·think · GLM-5·think |
| SuperCLUE 2026年3月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3 Flash·high · Grok 4.20·think · GLM-5·think |
| SuperCLUE 2026年3月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3 Flash·high · Grok 4.20·think · GLM-5·think |
| SuperCLUE 2026年3月 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3 Flash·high · Grok 4.20·think · GLM-5·think |
| SuperCLUE 2026年3月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3 Flash·high · Grok 4.20·think · GLM-5·think |
| SuperCLUE 2026年3月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年3月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.4·xhigh · Gemini 3 Flash·high · Grok 4.20·think · GLM-5·think |
| SuperCLUE 2026年5月 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.5·high · Claude Opus 4.8·high · Claude Opus 4.7·high · Gemini 3.5 Flash·high · Kimi K2.6·think · GLM-5.1·think |
| SuperCLUE 2026年5月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.5·high · Claude Opus 4.8·high · Claude Opus 4.7·high · Gemini 3.5 Flash·high · Kimi K2.6·think · GLM-5.1·think |
| SuperCLUE 2026年5月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.5·high · Claude Opus 4.8·high · Claude Opus 4.7·high · Gemini 3.5 Flash·high · Kimi K2.6·think · GLM-5.1·think |
| SuperCLUE 2026年5月 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.5·high · Claude Opus 4.8·high · Claude Opus 4.7·high · Gemini 3.5 Flash·high · Kimi K2.6·think · GLM-5.1·think |
| SuperCLUE 2026年5月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.5·high · Claude Opus 4.8·high · Claude Opus 4.7·high · Gemini 3.5 Flash·high · Kimi K2.6·think · GLM-5.1·think |
| SuperCLUE 2026年5月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Gemini 3.1 Pro·high · GPT-5.5·high · Claude Opus 4.8·high · Claude Opus 4.7·high · Gemini 3.5 Flash·high · Kimi K2.6·think · GLM-5.1·think |
| SuperCLUE 2026年7月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Qwen3.8 Max 0902 build·xhigh · DeepSeek V4.1 Flash·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GLM-5.3-Flash·max |
| SuperCLUE 2026年7月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Qwen3.8 Max 0902 build·xhigh · DeepSeek V4.1 Flash·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GLM-5.3-Flash·max |
| SuperCLUE 2026年7月 智能体任务规划 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体任务规划, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Qwen3.8 Max 0902 build·xhigh · DeepSeek V4.1 Flash·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GLM-5.3-Flash·max |
| SuperCLUE 2026年7月 智能体编程 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体编程, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Qwen3.8 Max 0902 build·xhigh · DeepSeek V4.1 Flash·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GLM-5.3-Flash·max |
| SuperCLUE 2026年7月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Qwen3.8 Max 0902 build·xhigh · DeepSeek V4.1 Flash·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GLM-5.3-Flash·max |
| SuperCLUE 2026年7月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Qwen3.8 Max 0902 build·xhigh · DeepSeek V4.1 Flash·max · GLM-5.3·max · Kimi K3·max · DeepSeek V4 Pro 0813 build·max · GLM-5.3-Flash·max |
| SuperCLUE-Terminal (Chinese agentic terminal coding) superclue-terminal | SuperCLUE-Terminal (CLUE team, Aug 2026; primary board/WeChat article NOT reached): Chinese-language real programming tasks of 50-110 interaction rounds each, all models through a Claude Code-based generic agent framework; numbers read from two secondary pages dated 14 Aug 2026 (we0.ai, news.aibase.com) that agree; cost = reported mean per-task cost in CNY (benchmark execution cost, not provider pricing); read 2026-10-20 ✓ | xmodel | Kimi K3·max · DeepSeek V4 Pro 0813 build·max |
| Surface Evolver bench yhenon-surface-evolver | yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓ | xmodel | Claude Fable 5·high · GPT-5.6 Sol·xhigh · GPT-5.5·high · Claude Opus 4.8·high · GPT-5.6 Terra·xhigh · GPT-5.5·medium · Gemini 3.8 Flash·high · Grok 4.5·high · Gemini 3.5 Flash·medium · Kimi K2.7 Code·solo · GLM-5.2·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro preview·high · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · GLM-5.3-Flash·max · Qwen3.8-27B·xhigh |
| surfevolver-pass yhenon-surface-evolver | — | xmodel | Claude Fable 5·high · GPT-5.6 Sol·xhigh · GPT-5.5·high · Claude Opus 4.8·high · GPT-5.6 Terra·xhigh · GPT-5.5·medium · Gemini 3.8 Flash·high · Grok 4.5·high · Gemini 3.5 Flash·medium · Kimi K2.7 Code·solo · GLM-5.2·high · DeepSeek V4.1 Flash·high · DeepSeek V4 Pro preview·high · GPT-5.6 Luna·medium · Claude Sonnet 5·medium · GLM-5.3-Flash·max · Qwen3.8-27B·xhigh |
| Surge AI · Antidote: Everyday Edition surgehq.ai | Surge AI Antidote: Everyday Edition leaderboard (surgehq.ai/benchmarks/antidote); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Claude Fable 5·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·medium · Gemini 3.5 Flash·medium · Kimi K3·max · Gemini 3.7 Flash·medium · GLM-5.3·max · Claude Fable 5.1·high · Claude Opus 5·high · Kimi K2.6·think · Claude Opus 4.7·high · Claude Opus 4.8·high · GPT-5.6 Sol·medium · GPT-5.5·medium · Claude Sonnet 4.6·high · Grok 4.20·think · Gemini 3.5 Flash-Lite·minimal · Grok 4.5·high · Grok 4.6·high · GPT-5.4 mini·nothink |
| Surge AI · Chartography surgehq.ai | Surge AI Chartography leaderboard (surgehq.ai/benchmarks/chartography); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Gemini 4 Argon·high · GPT-6 Astra·max · Claude Opus 5.5·max · GPT-6 Sol·max · Claude Fable 5.1·max · GPT-5.6 Sol·max · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Claude Fable 5·max · Gemini 3.6 Flash·medium · GPT-5.6 Terra·max · GPT-5.6 Luna·max · GPT-5.5·xhigh · GPT-5.4·xhigh · GPT-6 Luna·max · Claude Opus 5·max · Kimi K3·max · Gemini 3.1 Pro·high · Grok 4.6·xhigh · Grok 4.5·high · Claude Sonnet 5·max · Claude Opus 4.7·max · GLM-5.3-Flash·max · Claude Opus 4.8·max · Grok 4.7·xhigh · Grok 4.3·high · Kimi K2.6·think · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·minimal |
| Surge AI · ComplexConstraints surgehq.ai | Surge AI ComplexConstraints leaderboard (surgehq.ai/benchmarks/complex-constraints); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | GPT-6 Astra·max · Gemini 4 Argon·high · GPT-6 Sol·max · GPT-5.6 Sol·max · Claude Opus 5.5·max · DeepSeek V4.1 Flash·max · GPT-5.5·xhigh · Grok 4.7·xhigh · Gemini 3.8 Flash·high · GPT-5.4·xhigh · Claude Fable 5.1·max · Gemini 3.1 Pro·high · Gemini 3.7 Flash·high · Gemini 3.6 Flash·high · Claude Fable 5·max · Kimi K3·max · Claude Opus 5·max · Grok 4.6·xhigh · Grok 4.5·high · Claude Opus 4.8·max · GLM-5.3·max · Kimi K2.6·think · GLM-5.3-Flash·max · Gemini 3.5 Flash-Lite·high · Grok 4.20·think |
| Surge AI · DAYJOB: Finance surgehq.ai | Surge AI DAYJOB: Finance leaderboard (surgehq.ai/benchmarks/dayjob-finance); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Gemini 4 Argon·high · Claude Fable 5.1·max · Grok 4.7·xhigh · Grok 4.6·xhigh · Claude Opus 5·max · GPT-6 Sol·max · Claude Fable 5·max · GLM-5.3·max · GPT-5.6 Sol·max · Kimi K3·max · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · GLM-5.3-Flash·max · Claude Sonnet 5·max · GPT-6 Luna·max · GPT-5.6 Luna·max · GPT-5.6 Terra·xhigh |
| Surge AI · DAYJOB: Healthcare surgehq.ai | Surge AI DAYJOB: Healthcare leaderboard (surgehq.ai/benchmarks/dayjob-healthcare); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · GPT-6 Astra·max · Claude Fable 5.1·max · Gemini 4 Argon·high · Claude Opus 5·max · Grok 4.7·xhigh · Claude Fable 5·max · GPT-6 Sol·max · Grok 4.6·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·xhigh · Kimi K3·max · Gemini 3.8 Flash·high · GLM-5.3·max · GLM-5.3-Flash·max · GPT-5.6 Luna·max |
| Surge AI · EnterpriseBench: CoreCraft surgehq.ai | Surge AI EnterpriseBench: CoreCraft leaderboard (surgehq.ai/benchmarks/enterprisebench-corecraft); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5.5·max · Claude Fable 5·max · Claude Opus 5·max · Grok 4.7·xhigh · Grok 4.6·xhigh · Gemini 3.8 Flash·high · Gemini 3.7 Flash·high · Claude Opus 4.8·max · GLM-5.3·max · GPT-5.5·xhigh · Claude Sonnet 5·max · Gemini 3.5 Flash·high · GLM-5.3-Flash·max · GPT-5.6 Sol·max · GPT-6 Sol·max · GPT-6 Luna·max · Kimi K3·max · GPT-6 Astra·max · GPT-5.2·xhigh · DeepSeek V4.1 Flash·max · Gemini 3.6 Flash·high · GPT-5.4·xhigh · Claude Opus 4.7·max · Gemini 3.1 Pro·high · Kimi K2.6·think · Gemini 3 Flash·high · Claude Sonnet 4.6·high · Gemini 3 Pro·high |
| Surge AI · GDP.pdf surgehq.ai | Surge AI GDP.pdf leaderboard (surgehq.ai/benchmarks/gdp-pdf); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | GPT-6.1 Sol·max · GPT-6 Astra·max · Claude Sonnet 5.5·max · GPT-5.6 Sol·max · Claude Opus 5.5·max · Claude Fable 5·max · Gemini 4 Argon·high · Claude Fable 5.1·max · GPT-6 Sol·max · GPT-5.5·xhigh · Claude Opus 4.8·max · Claude Opus 5·max · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · GPT-6 Luna·max · Grok 4.7·xhigh · Claude Opus 4.7·max · DeepSeek V4.1 Flash·max · Kimi K3·max · Grok 4.6·xhigh · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · GLM-5.3-Flash·max · Grok 4.5·high · Kimi K2.6·think · Gemini 3.5 Flash-Lite·high · Gemini 3 Flash·high · Grok 4.3·high |
| Surge AI · GDP.xlsx surgehq.ai | Surge AI GDP.xlsx leaderboard (surgehq.ai/benchmarks/gdp-xlsx); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Gemini 4 Argon·high · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · GPT-6 Astra·max · Grok 4.7·xhigh · GPT-6.1 Sol·max · GLM-5.3·max · Gemini 3.8 Flash·high · GPT-6 Sol·max · Claude Sonnet 5·max · GPT-6 Luna·max · Kimi K3·max · Gemini 3.1 Pro·high |
| Surge AI · HANDBOOK.md Agents surgehq.ai | Surge AI HANDBOOK.md Agents leaderboard (surgehq.ai/benchmarks/handbook); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Claude Fable 5·max · Claude Opus 5·max · Grok 4.6·xhigh · GPT-5.6 Sol·max · Claude Opus 4.8·max · GPT-5.5·xhigh · Grok 4.5·high · Gemini 3.5 Flash·high · Gemini 3.7 Flash·high · GLM-5.3-Flash·max · Gemini 3.1 Pro·high · Kimi K2.6·think · Gemini 3.6 Flash·high · Gemini 3.5 Flash-Lite·high · Grok 4.3·high · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·max · Grok 4.7·xhigh · Gemini 4 Argon·high · GPT-6 Astra·max · GLM-5.3·max · Kimi K3·max · GPT-6.1 Sol·max · Gemini 3.8 Flash·high |
| Surge AI · Hemingway-bench surgehq.ai | Surge AI Hemingway-bench leaderboard (surgehq.ai/benchmarks/hemingway-bench); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Claude Fable 5·high · Gemini 3.7 Flash·medium · Claude Fable 5.1·high · Gemini 3.8 Flash·medium · Gemini 3.6 Flash·medium · Gemini 3.1 Pro·high · Gemini 3.5 Flash·medium · Kimi K3·max · GPT-5.6 Sol·medium · Gemini 3 Pro·high · GLM-5.3·max · Gemini 3 Flash·high · Claude Opus 5·high · Claude Opus 4.8·high · Claude Opus 4.7·high · GPT-5.5·medium · GLM-5.3-Flash·max · Kimi K2.6·think · Claude Sonnet 4.6·high · GPT-5.2 Chat (latest)·solo · Gemini 3.5 Flash-Lite·minimal · GPT-5.4·nothink · Grok 4.6·high · Grok 4.5·high · GPT-5.2·nothink |
| Surge AI · Riemann-bench surgehq.ai | Surge AI Riemann-bench leaderboard (surgehq.ai/benchmarks/riemann-bench); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | GPT-5.6 Sol·max · GPT-6 Astra·max · Claude Opus 5.5·max · Claude Opus 5·max · GPT-5.6 Terra·max · Claude Fable 5.1·max · GPT-6 Sol·max · Claude Fable 5·max · GPT-6 Luna·max · GPT-5.5·xhigh · Gemini 3.8 Flash·high · Claude Opus 4.8·max · GPT-5.4·xhigh · Gemini 3.7 Flash·high · Grok 4.5·high · GPT-5.2·xhigh · Kimi K3·max · Gemini 3.5 Flash·high · Gemini 3.1 Pro·high · Claude Opus 4.7·max · Gemini 3.6 Flash·high · DeepSeek V4.1 Flash·max · GLM-5.3·max · Gemini 3.5 Flash-Lite·high · GLM-5.3-Flash·max · Kimi K2.6·think |
| Surge AI · sudo L7 surgehq.ai | Surge AI sudo L7 leaderboard (surgehq.ai/benchmarks/sudo-l7); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · GPT-6 Astra·max · Claude Fable 5.1·max · GPT-6.1 Sol·max · Gemini 4 Argon·high · Grok 4.7·xhigh · Gemini 3.8 Flash·high |
| surge-twi-oct9 surge-twi | — | xmodel | Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Kimi K3·max · Claude Fable 5·max · GPT-5.6 Sol·max |
| SVGBench README leaderboard · score % svgbench-github | SVGBench (github.com/johnbean393/SVGBench) README leaderboard at commit 55c1cde8 (2026-02-12): single score column, 100+ SVG-generation questions with requirement-based scoring; effort = the label in parentheses where present; read 2026-10-20 ✓ | xmodel | GPT-5.2·xhigh · GPT-5.1·high · GPT-5.1·medium · Kimi K2 Thinking·solo · GPT-5.1 Chat (latest)·solo |
| SWE Atlas Fig. 14: cost per task vs Pass@1 on QnA + Test Writing scale-swe-atlas | SWE Atlas Appendix G Figure 14 (arXiv 2605.08366): per-trial token usage from the scaffold logs x each provider's published per-million-token API rates; 642 trials per model (372 Q&A + 270 Test Writing); native vendor scaffold; both axes DIGITISED from vector markers (axes calibrated on tick marks, residual 2e-5 pt); price date not given; read 2026-10-19 ✓ | xmodel | GPT-5.3-Codex·xhigh · Claude Opus 4.7·xhigh |
| SWE Bench Pro (xAI Grok 4.5 post) xai | SWE Bench Pro resolve rate %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; output tokens per task for Grok 4.5 and Opus 4.8 from the caption; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·max · GPT-5.5·xhigh |
| SWE Marathon (xAI Grok 4.5 post) xai | SWE Marathon resolution rate pass@1 %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · Claude Opus 4.7·max |
| SWE-bench Multilingual (S55) anthropic | Sonnet 5.5 card table 8.1.A · 3 models at max, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Sonnet 5·max · Claude Opus 5.5·max |
| SWE-bench Multimodal (S55) anthropic | Sonnet 5.5 card table 8.1.A · 3 models at max, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Sonnet 5·max · Claude Opus 5.5·max |
| SWE-bench Pro (cards) anthropic | Sonnet 5 card and Sonnet 5.5 card table 8.1.A · 5 Claude models at max, score only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5·max |
| SWE-Bench Pro (Public) (table) openai | SWE-Bench Pro (Public); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| SWE-Bench Pro V2 (Scale) scale-seal | Scale Labs board via Wayback 23 Sep 2026 · full set, Claude Code, one effort per model, score only, saturated ✓ | xmodel | Claude Opus 5·xhigh · Claude Fable 5.1·high · Claude Sonnet 5·xhigh · GPT-6 Astra·high · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·xhigh |
| SWE-Bench Pro V2 HARD (Scale) scale-seal | Scale Labs board via Wayback 23 Sep 2026 · HARD subset, Claude Code, score only ✓ | xmodel | Claude Opus 5·xhigh · Claude Fable 5.1·high · Claude Sonnet 5·xhigh · GPT-6 Astra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Sol·xhigh |
| SWE-bench Verified (GLM-4.7 card) zhipu | SWE-bench Verified; GLM-4.7 card 2026-01-29; settings: temperature 0.7 top-p 1.0 max new tokens 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| SWE-Marathon (GLM-5.2 card, evaluated by Abundant AI) zhipu | SWE-Marathon, 1M context 128K output; evaluation by Abundant AI; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · Gemini 3.1 Pro·high |
| swe-rebench 07/26 swe-rebench | ReAct minimal · Opus5/Fable5/Sonnet5 all at high ✓ | xmodel | Claude Fable 5·high · Grok 4.5·high · Claude Opus 5·high · GLM-5.2·high · GPT-5.6 Sol·medium · Claude Sonnet 5·high · GPT-5.6 Luna·medium · DeepSeek V4 Pro preview·high |
| synthbench-althing-globalopinionqa github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·nothink · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·high · DeepSeek V4.1 Flash·high · Gemini 3.5 Flash-Lite·minimal · GPT-6 Luna·medium |
| synthbench-althing-gss github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·nothink · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·high · DeepSeek V4.1 Flash·high · Gemini 3.5 Flash-Lite·minimal · GPT-6 Luna·medium |
| synthbench-althing-opinionsqa github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·nothink · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·high · DeepSeek V4.1 Flash·high · Gemini 3.5 Flash-Lite·minimal · GPT-6 Luna·medium |
| synthbench-althing-subpop github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·nothink · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·high · DeepSeek V4.1 Flash·high · Gemini 3.5 Flash-Lite·minimal · GPT-6 Luna·medium |
| synthbench-plain-globalopinionqa github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·low · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·low · DeepSeek V4.1 Flash·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low · GPT-6 Luna·nothink · Qwen3.8 Flash·nothink |
| synthbench-plain-gss github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·low · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·low · DeepSeek V4.1 Flash·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low · GPT-6 Luna·nothink · Qwen3.8 Flash·nothink |
| synthbench-plain-opinionsqa github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·low · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·low · DeepSeek V4.1 Flash·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low · GPT-6 Luna·nothink · Qwen3.8 Flash·nothink |
| synthbench-plain-subpop github-synthbench | — | xmodel | Claude Haiku 4.5·nothink · Claude Haiku 5.5·low · Claude Sonnet 4.6·nothink · Claude Sonnet 5.5·low · DeepSeek V4.1 Flash·low · Gemini 3.5 Flash-Lite·minimal · Gemini 3.8 Flash·low · GPT-6 Luna·nothink · Qwen3.8 Flash·nothink |
| Synthorai (ja mirror): 4-question tool-reading loop, Sonnet 5.5 (sweep) and Opus 5.5 default synthorai | native Messages API via gateway, 4 tasks x3 reps per row (n=12), cost computed from list price; qiita.com/Synthorai/items/98e776a66f483f3ceeaf (mirror of synthorai.io/ja/blog/claude-sonnet-5-5-cost/); read 2026-10-05 ✓ | xmodel | Claude Sonnet 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·max · Claude Sonnet 5·high · GPT-6.1 Sol·medium |
| synthorai-fix30-notools synthorai | — | xmodel | DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low |
| synthorai-flash1017-fix30 synthorai | — | xmodel | DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max |
| synthorai-oneshot13 synthorai | — | xmodel | DeepSeek V4.1 Flash 1017 build·nothink · DeepSeek V4.1 Flash 1017 build·low · DeepSeek V4.1 Flash 1017 build·high · DeepSeek V4.1 Flash 1017 build·max |
| Takuya__ (qiita, item 2a513efa) 12 sqlite-utils bug-fix/feature tasks (incl 4 real upstream issues), x2 runs, full test suite qiita-takuya55 | Claude Code 2.1.293, Read/Edit/Write/Grep/Glob/Bash, pass = all hidden+existing tests (~1000) green; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Sonnet 5.5·medium |
| Takuya__ (qiita, item 2a513efa) Banking77 intent classification + "narrow candidates first" prompt addition qiita-takuya55 | same task as qiita-takuya55-banking77 but with an added system-prompt instruction to narrow candidates first: different configuration; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium |
| Takuya__ (qiita, item 2a513efa) Banking77 intent classification, 154 msgs/77 labels, no tools, single API call qiita-takuya55 | no-tools system-prompt label list, single API call per message, cost computed from measured tokens x price table per 1,000 msgs; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·max · Claude Sonnet 5.5·medium |
| Takuya__ (qiita, item 2a513efa) code-reading Q&A, sqlite-utils 3.38, 9 questions x2 runs qiita-takuya55 | Claude Code 2.1.293 claude -p stream-json, Read/Grep/Glob/Bash only, cost = measured tokens x official price table; read 2026-10-08 ✓ | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium |
| Takuya__ (qiita, item 2a513efa) Sonnet-lead orchestration, 9 Q x3 runs, pure single-model baselines only qiita-takuya55 | Claude Code 2.1.293, Explore subagent delegation; only the two PURE single-model runs are rows (mixed Sonnet-lead+Haiku-Explore run excluded: not a model x effort couple); read 2026-10-08 ✓ | xmodel | Claude Sonnet 5.5·medium · Claude Haiku 5.5·medium |
| TashonBraganca/messy-docs-bench: field-extraction accuracy, Opus 5.5 vs Sonnet 5 github-messydocsbench | 137 messy documents (receipts/invoices/scans/W-2/statements/contracts), field-level extraction accuracy; github.com/TashonBraganca/messy-docs-bench, src/messydocs_summary.json; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5·high · GPT-5.6 Terra·medium |
| TasteVal compute multiplier vs expert baseline (Researcher models, 8 ML-research tasks) arxiv-2610.06824 | 8 uncontaminated AI R&D tasks, 6 seeds/model/task, 40 H100-hours; Researcher model + fixed Coder (Opus 4.8); Table 2 of arXiv 2610.06824; score only (cost in round 2); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 4.8·max · GLM-5.3·max · GPT-5.6 Sol·max · Claude Opus 4.7·max · GPT-5.5·xhigh · GPT-5.2·high · GPT-5.4·xhigh |
| TasteVal performance multiplier vs expert baseline arxiv-2610.06824 | 8 uncontaminated AI R&D tasks, 6 seeds/model/task, 40 H100-hours; Researcher model + fixed Coder (Opus 4.8); Table 2 of arXiv 2610.06824; score only (cost in round 2); read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · GPT-6 Astra·max · Claude Opus 4.8·max · GLM-5.3·max · GPT-5.6 Sol·max · Claude Opus 4.7·max · GPT-5.5·xhigh · GPT-5.2·high · GPT-5.4·xhigh |
| tau-bench 2 airline (text) leaderboard github-sierra-tau2-bench | tau2-bench v1.0.1 airline domain, text modality, pass^1 (%) over 4 trials, user simulator gpt-5.2 low reasoning, standard tau-bench scaffold, submitted by Sierra Research (Feb-May 2026), board head 989d328 of 2026-10-20; read 2026-10-20 ✓ | xmodel | GPT-5.2·high · GPT-5.2·nothink · Gemini 3 Flash·high · Gemini 3 Pro·high · GLM-5·think |
| tau-bench 2 retail (text) leaderboard github-sierra-tau2-bench | tau2-bench v1.0.1 retail domain, text modality, pass^1 (%) over 4 trials, user simulator gpt-5.2 low reasoning, standard tau-bench scaffold, submitted by Sierra Research (Feb-May 2026), board head 989d328 of 2026-10-20; read 2026-10-20 ✓ | xmodel | GPT-5.2·high · GPT-5.2·nothink · Gemini 3 Flash·high · Gemini 3 Pro·high · GLM-5·think |
| tau-bench 2 telecom (text) leaderboard github-sierra-tau2-bench | tau2-bench v1.0.1 telecom domain, text modality, pass^1 (%) over 4 trials, user simulator gpt-5.2 low reasoning, standard tau-bench scaffold, submitted by Sierra Research (Feb-May 2026), board head 989d328 of 2026-10-20; read 2026-10-20 ✓ | xmodel | GPT-5.2·high · GPT-5.2·nothink · Gemini 3 Flash·high · Gemini 3 Pro·high · GLM-5·think |
| tau2-Bench (GLM-4.7 card) zhipu | tau2-Bench; GLM-4.7 card 2026-01-29; settings: temperature 0 max new tokens 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| Tau2-bench Telecom (OpenAI GPT-5.5 post chart) openai | Tau2-bench Telecom; OpenAI GPT-5.5 launch post chart per reasoning effort (none..xhigh); no cost on this chart (tokens/tool calls only); published 2026-04-23 (snapshot Wayback 20260423180657); read 2026-10-19 ✓ | xmodel | GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh |
| Tau2-bench Telecom*** (original prompts) (table) openai | Tau2-bench Telecom*** (original prompts); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| TaxCalcBench TY25, no web search (column-tax/tax-calc-bench) column-tax-tax-calc-bench | 50 TY25 federal+state tax-return cases, 1 run each; strict-correct-return % is the score (lenient% also in the evidence but not used here per instructions); cost = mean litellm-estimate USD/return (n priced noted when <50); source: taxcalc_ty25_claude_by_effort.txt, re-verified against raw evaluation_result_*.md for opus-5.5/ultrathink (40.00%/50.00%/$1.135, exact match); read 2026-10-01 ✓ | xmodel | Gemini 3.7 Flash·high · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Gemini 3.5 Flash·minimal · GPT-6 Astra·max · Claude Opus 5.5·max · Claude Fable 5.1·max · GPT-6.1 Sol·max · Claude Opus 5.5·high · GPT-6 Astra·high · Claude Opus 5.5·xhigh · GPT-6 Astra·medium · GPT-6.1 Sol·high · GPT-6 Astra·low · GPT-5.6 Sol·max · Claude Fable 5·xhigh · GPT-6 Sol·max · GPT-5.6 Sol·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · GPT-5.5·high · GPT-6.1 Sol·low · Claude Fable 5·high · GPT-6.1 Sol·medium · Claude Opus 5.5·medium · GPT-5.6 Sol·medium · Claude Fable 5.1·medium · Claude Opus 5·xhigh · GPT-6 Sol·high · GPT-5.5·xhigh · Claude Opus 4.8·xhigh · Claude Sonnet 5.5·xhigh · Claude Opus 5·max · Claude Opus 5·high · Claude Fable 5·medium · GPT-6 Sol·medium · Claude Opus 5·medium · GPT-6 Luna·max · GPT-5.5·medium · Claude Fable 5.1·low · Claude Sonnet 5.5·high · Claude Opus 4.8·medium · Claude Opus 5.5·low · Claude Opus 4.8·high · Claude Fable 5·low · GPT-5.6 Sol·low · Gemini 3.6 Flash·high · Claude Opus 5·low · GPT-6 Luna·medium · Kimi K3·max · GPT-6 Sol·low · GPT-5.5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Gemini 3.6 Flash·medium · Gemini 3.8 Flash·high · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · GPT-6 Luna·high · Claude Sonnet 5·xhigh · Gemini 3.8 Flash·medium · GPT-5.5·nothink · Gemini 3.6 Flash·minimal · Claude Haiku 5.5·high · Claude Haiku 5.5·medium · GPT-6 Luna·low · GPT-6 Sol·nothink · Gemini 3.6 Flash·low · GPT-5.6 Sol·nothink · Claude Opus 4.8·low · Gemini 3.8 Flash·low · Claude Haiku 5.5·low · GPT-6 Luna·nothink |
| TaxCalcBench TY25, with web search (column-tax/tax-calc-bench) column-tax-tax-calc-bench | Same as taxcalc-ty25-notool but with the web-search tool enabled; read 2026-10-01 ✓ | xmodel | Gemini 3.7 Flash·medium · Gemini 3.7 Flash·low · GPT-6 Sol·max · GPT-6.1 Sol·high · Claude Opus 5.5·max · GPT-6.1 Sol·max · GPT-5.6 Sol·max · GPT-6 Astra·medium · GPT-6 Astra·max · GPT-6 Astra·high · GPT-5.5·xhigh · GPT-6.1 Sol·medium · GPT-6 Astra·low · GPT-5.5·medium · GPT-5.6 Sol·medium · GPT-5.5·high · Claude Fable 5.1·xhigh · GPT-5.6 Sol·high · Claude Opus 5.5·xhigh · GPT-6.1 Sol·low · Claude Fable 5.1·max · GPT-6 Sol·high · GPT-6 Sol·medium · Claude Opus 5·max · Claude Sonnet 5.5·max · Claude Fable 5.1·high · Claude Fable 5·max · Claude Opus 5.5·high · Claude Sonnet 5.5·xhigh · Kimi K3·max · Claude Fable 5·xhigh · Claude Opus 5·xhigh · Claude Opus 4.8·xhigh · GPT-6 Luna·max · GPT-5.6 Sol·low · Claude Opus 5.5·medium · Claude Fable 5·high · GPT-6 Sol·low · Claude Opus 5·high · Claude Fable 5.1·medium · Claude Sonnet 5.5·high · Claude Opus 4.8·max · Claude Haiku 5.5·xhigh · GPT-6 Luna·medium · Gemini 3.8 Flash·low · Gemini 3.6 Flash·high · Claude Fable 5·medium · Claude Fable 5·low · GPT-5.6 Sol·nothink · Claude Fable 5.1·low · Claude Opus 5·medium · Gemini 3.6 Flash·medium · Claude Opus 4.8·high · GPT-6 Sol·nothink · Claude Opus 5.5·low · GPT-6 Luna·high · Claude Opus 4.8·medium · Claude Haiku 5.5·high · GPT-5.5·low · Gemini 3.6 Flash·low · GPT-6 Luna·low · Claude Opus 5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·low · Claude Haiku 5.5·medium · GPT-5.5·nothink · GPT-6 Luna·nothink · Claude Haiku 5.5·low · Gemini 3.6 Flash·minimal |
| TB-Science 0.1 (S55) anthropic | Sonnet 5.5 card p114 · Claude Code --bare, 5 models at max, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Opus 5·max · Claude Fable 5·max |
| tb21cc tbench | — | xmodel | Claude Opus 4.8·high · Claude Sonnet 5·high |
| TCSAlgBench 10-round discussion acceptance % (calibration runs) arxiv-2609.35606 | TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); separate runs of Table 9 (acceptance %, RMS-CE reported separately); read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·high · GPT-5.5·xhigh · GPT-5.5·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·high · Claude Opus 4.8·max |
| TCSAlgBench 10-round discussion, 5-run coverage arxiv-2609.35606 | TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); 10-round discussion, coverage over 5 runs; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · Claude Fable 5·high · Claude Fable 5·xhigh |
| TCSAlgBench direct inference, 10-run coverage arxiv-2609.35606 | TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); direct inference, coverage = challenges accepted in at least one of 10 runs; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·high · GPT-5.5·xhigh · GPT-5.5·high · Claude Opus 4.8·max · Claude Opus 4.8·xhigh · Claude Opus 4.8·high · Claude Fable 5·xhigh · Claude Fable 5·high |
| techkwon-token-diet-lesson github-techkwon | — | xmodel | Claude Fable 5.2·high · Claude Fable 5.2·low · Claude Opus 5.5·high |
| techkwon-token-diet-motion github-techkwon | — | xmodel | Claude Fable 5.2·high · Claude Fable 5.2·low · Claude Opus 5.5·high |
| Terminal Bench 2.0 (GLM-4.7 card) zhipu | Terminal Bench 2.0; GLM-4.7 card 2026-01-29; settings: temperature 0.7 top-p 1.0 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| Terminal Bench 2.1 (xAI Grok 4.5 post) xai | Terminal Bench 2.1 score %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · Claude Opus 4.8·max · Claude Opus 4.7·max |
| Terminal Bench Hard (GLM-4.7 card) zhipu | Terminal Bench Hard; GLM-4.7 card 2026-01-29; settings: temperature 0.7 top-p 1.0 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo · GPT-5.1·high |
| Terminal-Bench 2.0 (OpenAI GPT-5.5 post chart) openai | Terminal-Bench 2.0; OpenAI GPT-5.5 launch post chart per reasoning effort (none..xhigh); no cost on this chart (tokens/tool calls only); published 2026-04-23 (snapshot Wayback 20260423180657); read 2026-10-19 ✓ | xmodel | GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh |
| Terminal-Bench 2.1 (OpenAI GPT-5.6 post chart) openai | Terminal-Bench 2.1 accuracy, code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Terra·max · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Luna·max · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh |
| Terminal-Bench 3.0 (GLM-5.3 card) zhipu | Terminal-Bench-3, Claude Code 2.1.207, avg@3, 600 turns, 10h timeout, Tool Search disabled, official verifier; GLM-5.3 card 2026-10-14; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · Claude Fable 5·max |
| Terminal-Bench 4.0 tbench | tbench.ai primary payload · 66 tasks × 330 trials, Claude Code, all at max; cost basis undocumented ✓ | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Fable 5.1·max · GPT-6 Astra·xhigh · GPT-6 Astra·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·high · GPT-6 Astra·medium · Claude Fable 5.1·medium · Claude Opus 5·xhigh · Claude Opus 5·max · GPT-6 Astra·low · Claude Opus 5·high · GPT-6 Sol·max · Claude Opus 5·medium · Claude Fable 5·max · Claude Fable 5.1·low · Grok 4.7·xhigh · Claude Opus 5·low · Claude Opus 4.8·max · Grok 4.6·high · GPT-6 Luna·max · Grok 4.5·high · Claude Sonnet 5·max |
| Terminal-Bench 4.0 (OpenAI GPT-6 Astra post chart, code mode) openai | Terminal-Bench 4.0; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = code; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| Terminal-Bench 4.0 (OpenAI GPT-6 Astra post chart, non-code mode) openai | Terminal-Bench 4.0; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Claude Fable 5·max · Claude Opus 5·max · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · GPT-5.6 Sol·max |
| Terminal-Bench 4.0 (xAI Grok 4.7 post) xai | Terminal-Bench 4.0 'multi-hour terminal work' %, harness not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Grok 4.7·xhigh · Grok 4.6·high · GPT-5.6 Sol·max · Claude Fable 5.1·max |
| Terminal-Bench Science 0.1 (OpenAI GPT-6 Astra post chart, code mode) openai | Terminal-Bench Science 0.1; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = code; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max |
| Terminal-Bench Science 0.1 (OpenAI GPT-6 Astra post chart, non-code mode) openai | Terminal-Bench Science 0.1; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | GPT-5.6 Sol·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 5·max |
| Terminal-Bench Science 0.1 (OpenAI GPT-6.1 Sol post) openai | Terminal-Bench Science 0.1; OpenAI GPT-6.1 Sol post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; Wayback first capture; read 2026-10-19 ✓ | xmodel | Claude Opus 5.5·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-6.1 Terra·low · GPT-6.1 Terra·medium · GPT-6.1 Terra·high · GPT-6.1 Terra·xhigh · GPT-6.1 Terra·max |
| Terminal-Bench v3.0 (xAI Grok 4.6 post) xai | Terminal-Bench v3.0 %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Grok 4.6·high · Grok 4.5·high · GPT-5.6 Sol·max · Claude Fable 5·max |
| Test Cabinet · claude 2.1.203 · shatter v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.203; case shatter v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Haiku 4.5·nothink · Claude Opus 4.8·nothink |
| Test Cabinet · claude 2.1.204 · shatter v1.0.0 · warhead testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.204; case shatter v1.0.0; variant warhead; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Haiku 4.5·nothink |
| Test Cabinet · claude 2.1.205 · fathom v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.205; case fathom v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Sonnet 4.6·nothink · Claude Haiku 4.5·nothink |
| Test Cabinet · claude 2.1.206 · floe-bear v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.206; case floe-bear v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Haiku 4.5·nothink |
| Test Cabinet · claude 2.1.208 · fathom v1.1.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.208; case fathom v1.1.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Haiku 4.5·nothink |
| Test Cabinet · claude 2.1.211 · valence v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.211; case valence v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Haiku 4.5·nothink |
| Test Cabinet · claude 2.1.233 · deepcore v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.233; case deepcore v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Opus 5·high |
| Test Cabinet · claude 2.1.233 · fathom v2.0.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.233; case fathom v2.0.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Opus 5·high |
| Test Cabinet · claude 2.1.234 · arc-foundry v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case arc-foundry v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · Claude Opus 4.8·nothink |
| Test Cabinet · claude 2.1.234 · deepcore v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case deepcore v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · Claude Opus 4.8·nothink |
| Test Cabinet · claude 2.1.234 · pong v2.1.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case pong v2.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Opus 5·high |
| Test Cabinet · claude 2.1.235 · coil v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case coil v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Opus 4.8·nothink · Claude Opus 5·high |
| Test Cabinet · claude 2.1.235 · pong v2.1.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case pong v2.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Fable 5·high · Claude Sonnet 4.6·nothink · Claude Sonnet 5·high · Claude Haiku 4.5·nothink · Claude Opus 4.8·nothink |
| Test Cabinet · claude 2.1.235 · pong v2.1.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case pong v2.1.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Fable 5·high · Claude Opus 4.8·nothink · Claude Opus 5·high |
| Test Cabinet · claude 2.1.237 · band-of-bots v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.237; case band-of-bots v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Fable 5·high · Claude Sonnet 5·high |
| Test Cabinet · claude 2.1.237 · pong v2.1.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.237; case pong v2.1.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Sonnet 4.6·nothink · Claude Opus 5·high |
| Test Cabinet · claude 2.1.238 · band-of-bots v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.238; case band-of-bots v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Fable 5·high · Claude Sonnet 5·high |
| Test Cabinet · claude 2.1.302 · pong v3.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.302; case pong v3.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·high · Claude Sonnet 5.5·high |
| Test Cabinet · codex 0.142.4 · pong v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.142.4; case pong v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.4 mini·medium |
| Test Cabinet · codex 0.142.4 · pong v1.0.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.142.4; case pong v1.0.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.4 mini·medium |
| Test Cabinet · codex 0.142.5 · shatter v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.142.5; case shatter v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4 mini·medium · GPT-5.5·medium |
| Test Cabinet · codex 0.143.0 · fathom v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.143.0; case fathom v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4 mini·medium · GPT-5.5·medium |
| Test Cabinet · codex 0.144.0 · fathom v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.144.0; case fathom v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.144.1 · floe-bear v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.144.1; case floe-bear v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Terra·medium · GPT-5.6 Sol·low · GPT-5.6 Luna·medium · GPT-5.5·medium |
| Test Cabinet · codex 0.144.3 · fathom v1.1.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.144.3; case fathom v1.1.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.4 mini·medium · GPT-5.6 Sol·low · GPT-5.6 Luna·medium · GPT-5.5·medium · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.144.4 · fathom v1.1.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.144.4; case fathom v1.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.144.4 · fathom v1.1.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.144.4; case fathom v1.1.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium · GPT-5.4 mini·medium |
| Test Cabinet · codex 0.144.4 · floe v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.144.4; case floe v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.6 Sol·low |
| Test Cabinet · codex 0.145.0 · pong v2.0.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.145.0; case pong v2.0.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Luna·medium |
| Test Cabinet · codex 0.147.0 · arc-foundry v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case arc-foundry v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · band-of-bots v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case band-of-bots v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · coil v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case coil v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · comfort-zone v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case comfort-zone v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · dealers-choice v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case dealers-choice v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.6 Sol·low · GPT-5.5·medium |
| Test Cabinet · codex 0.147.0 · deepcore v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case deepcore v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · fathom v2.0.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case fathom v2.0.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · floe v2.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case floe v2.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Luna·medium |
| Test Cabinet · codex 0.147.0 · outside-the-box v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case outside-the-box v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.6 Sol·low |
| Test Cabinet · codex 0.147.0 · pong v2.0.1 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case pong v2.0.1; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.4 mini·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · pong v2.1.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case pong v2.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · pong v2.1.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case pong v2.1.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · shatter v2.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case shatter v2.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.4 mini·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · shatter v2.0.1 · warhead testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case shatter v2.0.1; variant warhead; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.4·medium · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium |
| Test Cabinet · codex 0.147.0 · well-well-well v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.147.0; case well-well-well v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · GPT-5.6 Luna·medium |
| Test Cabinet · codex 0.162.0 · pong v3.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.0; case pong v3.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·low · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · cascade v3.0.0 · draw-three testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case cascade v3.0.0; variant draw-three; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · coil v2.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case coil v2.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · facet v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case facet v1.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · fathom v3.0.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case fathom v3.0.0; variant kindle; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · kessler v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case kessler v1.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·low · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · pong v3.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case pong v3.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium · GPT-6 Luna·medium · GPT-6 Sol·medium |
| Test Cabinet · codex 0.162.1 · pong v3.0.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case pong v3.0.0; variant gyre; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium · GPT-6 Astra·low · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · refract v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case refract v1.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.162.1 · shatter v3.0.0 · warhead testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.162.1; case shatter v3.0.0; variant warhead; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Sol·low · GPT-5.6 Terra·medium · GPT-6 Astra·low · GPT-6 Luna·medium · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.163.0 · kessler v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.163.0; case kessler v1.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.163.0 · orrery v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.163.0; case orrery v1.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.163.0 · volute v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.163.0; case volute v1.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| Test Cabinet · codex 0.163.0 · wick v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness codex 0.163.0; case wick v1.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | GPT-5.6 Sol·low · GPT-6 Astra·low · GPT-6 Sol·medium · GPT-6.1 Sol·low |
| tfcxzaqwerds: 9 hard reasoning questions x3 runs (no tools), score note-tfcxzaqwerds | note.com/tfcxzaqwerds/n/nab4b132711a2; same Claude Code headless setup, effort high, no tools, maze/24-digit-multiplication/logic-puzzles, 9 questions x3 runs = 27 trials/model, 15-min time cap per question; read 2026-10-01 ✓ | xmodel | Claude Sonnet 5·high · Claude Sonnet 5.5·high |
| The Endless Exam (2609.24555) — 69 open-ended maths constructions, Claude Code tools off arxiv-2609.24555 | Table 2, 69 instances, overall score = 100 x mean relative quality vs reference; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | GPT-5.6 Luna·medium · GPT-5.6 Luna·high · DeepSeek V4.1 Flash·low · DeepSeek V4.1 Flash·high · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Opus 5.5·medium · Claude Opus 5.5·high · GPT-6 Astra·medium · GPT-6 Astra·high |
| Thinkingbox-bench pass@1 (507 tasks x20 trials, 5 stateful business domains) arxiv-2608.19741 | Thinkingbox-bench 507 tasks, N=20 trials, own MCP sandbox, simulated user GPT-5.4-mini, Azure-hosted API (local vLLM for Qwen), published 2026-08-19; cost = campaign total USD over 507x20 trials; read 2026-10-20 ✓ | xmodel | Claude Opus 5·medium · GPT-5.4·medium · GPT-5.6 Sol·medium · GPT-6 Astra·medium · GPT-5.2·medium · Grok 4.3·medium |
| ThreadKeeper: summarise a 128k-token fake chat (facts, threads, inventions), composite v2 github-threadkeeper | neko-legends/threadkeeper results/leaderboard.csv; 128k corpus seed 12801; Claude Code 2.1.289; judge venice/gemini-3-8-flash (neutral family) for every arm; composite = 100*(0.45 facts + 0.30 threads + 0.25 invention_score); subscription, no cost; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·medium · Claude Sonnet 5.5·medium · Claude Opus 5.5·max |
| Tier-2 engineering solve items (of 25), Claude Code CLI no tools arxiv-2610.06668 | 25 multi-regime engineering solve problems, text only, Claude Code CLI replacing the system prompt, effort high where exposed; Supp. Table 3 of arXiv 2610.06668; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Sonnet 5·high · Claude Opus 4.8·high · GPT-6 Sol·high · GPT-5.6 Sol·high · GPT-5.5·high |
| Timeline-Bench (video editing), vendor-native harnesses (Codex CLI / Claude Code) arxiv-2609.35143 | Timeline-Bench: same 56 tasks, 1 run per task; harness = developer's own (GPT in Codex CLI 0.155.1, Claude in Claude Code 2.1.278); score = Res. %; cost = Table 7 mean model cost per run (Claude Code recorded; Codex CLI estimated by the authors from tokens at base-rate prices); curated-guidance and computer-use Astra variants not rows (different setups); read 2026-10-20 ✓ | xmodel | Claude Opus 5·max · GPT-6 Astra·max · Claude Fable 5.1·max · GPT-5.6 Sol·max |
| Tinybird SQL benchmark llm-benchmark.tinybird.live | Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓ | xmodel | Claude Opus 4.7·nothink · Claude Fable 5.1·high · Claude Opus 5.5·high · Claude Sonnet 5.5·high · Claude Opus 5·high · Claude Sonnet 4.6·nothink · Claude Haiku 5.5·nothink · Grok 4.6·high · Gemini 3.6 Flash·medium · GLM-5.3-Flash·max · DeepSeek V4.1 Flash·high · Gemini 4 Argon·medium · GLM-5.2·high · Grok 4.7·high · Gemini 3.7 Flash·medium · Grok 4.5·high · Qwen3.8 Max·xhigh · Gemini 3.8 Flash·medium · GLM-5.3-FlashX·max · GPT-5.6 Sol·medium · GPT-6 Luna·medium · DeepSeek V4 Flash 0731 build·high · Kimi K2.7 Code·solo · Gemini 3.5 Flash-Lite·minimal · GPT-6.1 Sol·medium · GPT-5.6 Luna·medium · GPT-6 Luna Pro·medium · GPT-6 Sol·medium · GPT-5.6 Terra·medium |
| TLA-Verus tlaverus_algo 40 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 12: tlaverus_algo 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·medium · Grok 4.5·high · Grok 4.3·high · GLM-5.2·high |
| TLA-Verus tlaverus_llm 40 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 13: tlaverus_llm 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·medium · Grok 4.5·high · Grok 4.3·high · GLM-5.2·high |
| TLA-Verus verusbench 150 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 11: verusbench 150 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Claude Opus 5·high · GPT-5.6 Terra·medium · Grok 4.5·high · Grok 4.3·high · GLM-5.2·high |
| tokensave-brickbreaker-cost tokensave | — | xmodel | Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high |
| tokensave-brickbreaker-cost-first tokensave | — | xmodel | Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high |
| tokensave-vibe-2048 tokensave | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| tokensave-vibe-brick tokensave | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| tokensave-vibe-sudoku tokensave | — | xmodel | Claude Haiku 5.5·medium · Claude Sonnet 5.5·medium · Claude Opus 5.5·medium |
| Tool-Decathlon (GLM-5.2 card) zhipu | Tool-Decathlon official evaluation service, max_token 128K; GLM-5.2 card; read 2026-10-19 ✓ | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · Gemini 3.1 Pro·high |
| Toolathlon (table) openai | Toolathlon; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh |
| Toolathlon previous leaderboard (pre-Verified snapshot, 51 models) toolathlon | Toolathlon archived 'Previous Toolathlon leaderboard' (snapshot before Toolathlon-Verified, 51 models; task definitions differ so not comparable with the current table), Pass@1 %; badge = evaluated by the Toolathlon team, no badge = copied from a vendor post; fetched 2026-10-20; cells identical to held cells dropped; read 2026-10-20 ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4·xhigh · GPT-5.3-Codex·xhigh · GPT-5.2·xhigh · GPT-5.4 mini·xhigh · GPT-5.2·high · GPT-5.1·high · GPT-5.4 nano·xhigh |
| Toolathlon-Verified anthropic | Sonnet 5.5 card table 8.14.5.A · 7 Claude models at max, Pass@1, score-only ✓ | xmodel | Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Opus 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max |
| Toolathlon-Verified leaderboard (current, 108 tasks) toolathlon | Toolathlon-Verified (released 30 Jun 2026), 108 tasks, 32 MCP servers / 604 tools + 7 local toolkits, Pass@1 % (+-CI) over 3 runs; rows carry the 'Evaluated by us' badge unless noted; Agent 'Default'; page fetched 2026-10-20 (newest row 2026-08-30); cells identical to held cells (vendor cards/blogs copying the board) dropped; read 2026-10-20 ✓ | xmodel | GLM-5.3-Flash·max · Kimi K3·max · Claude Opus 4.8·max · DeepSeek V4 Pro 0813 build·max · GPT-5.5·xhigh · Claude Sonnet 5·max · Gemini 3.5 Flash·high · Gemini 3.1 Pro·high · Kimi K2.7 Code·solo · Gemini 3.5 Flash-Lite·high |
| Towards AI internal writing-Elo board, Sep 22, 2026 reddit-1wr03jc-img1 | 10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; top-16 leaderboard bar chart, footer 'Board of Sep 22, 2026'; values are printed bar labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 5·max · DeepSeek V4.1 Flash·max · GPT-6 Sol·max · Claude Opus 4.8·max · Grok 4.7·high · Claude Sonnet 5·max |
| tracker-quality-score github-startakovsky-llm-model-tracker | — | xmodel | Kimi K2.7 Code·solo |
| TSNBench MCQA accuracy (Time-Sensitive Networking, 939 questions) tsnbench-2605.09481 | TSNBench Table 4: 939 MCQA questions on Time-Sensitive Networking, temperature 0.0 (dagger models: default settings), 3 runs for consistency; accuracy (consistency/latency/ECE/Brier/CW rate = same runs, rule 8); arXiv 2605.09481 (May 2026); read 2026-10-20 ✓ | xmodel | Grok 4.1 Fast·nothink |
| tuabench-cc arxiv-2606.28480 | — | xmodel | Claude Opus 4.8·max · GPT-5.5·xhigh · GPT-5.5·high · GPT-5.5·medium · Claude Opus 4.7·xhigh · GPT-5.5·low · GPT-5.5·nothink |
| tycho-arc3 arxiv-2607.28287 | — | xmodel | Claude Opus 4.8·xhigh · Claude Opus 5·xhigh · GPT-5.6 Sol·max |
| UGI Leaderboard — UGI composite (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, composite UGI score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Claude Sonnet 4.6·low · Claude Sonnet 4.6·high · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Kimi K2 Thinking·solo · GPT-5.1 Chat (latest)·solo · GPT-5.1·low · GPT-5.1·medium · GPT-5.1·high · Gemini 3 Pro·high · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · GLM-4.7·think · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Qwen3.5-122B-A10B·nothink · GPT-5.2·low · GPT-5.2·nothink · GPT-5.2·medium · GPT-5.2·high · GPT-5.2 Chat (latest)·solo · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · Grok 4.20·nothink · Grok 4.20·think · Qwen3.6-27B·nothink · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Grok 4.5·high · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · Grok 4.6·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · Grok 4.7·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh |
| UGI Leaderboard — World Model factual recall (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, recipe/geoguessr/weight/music percent-error battery; test dates vary by model; read 2026-10-05 ✓ | xmodel | Claude Sonnet 4.6·low · Claude Sonnet 4.6·high · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Kimi K2 Thinking·solo · GPT-5.1 Chat (latest)·solo · GPT-5.1·low · GPT-5.1·medium · GPT-5.1·high · Gemini 3 Pro·high · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · GLM-4.7·think · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Qwen3.5-122B-A10B·nothink · GPT-5.2·low · GPT-5.2·nothink · GPT-5.2·medium · GPT-5.2·high · GPT-5.2 Chat (latest)·solo · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · Grok 4.20·nothink · Grok 4.20·think · Qwen3.6-27B·nothink · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Grok 4.5·high · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · Grok 4.6·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · Grok 4.7·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh |
| UGI Leaderboard — Writing quality (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, LLM-judged Writing score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Claude Sonnet 4.6·low · Claude Sonnet 4.6·high · Claude Opus 4.7·low · Claude Opus 4.7·medium · Claude Opus 4.7·high · Claude Opus 4.7·max · Claude Opus 4.8·low · Claude Opus 4.8·medium · Claude Opus 4.8·high · Claude Opus 4.8·xhigh · Claude Opus 4.8·max · Claude Fable 5·low · Claude Fable 5·medium · Claude Fable 5·high · Claude Fable 5·xhigh · Claude Fable 5·max · Claude Sonnet 5·low · Claude Sonnet 5·medium · Claude Sonnet 5·high · Claude Sonnet 5·xhigh · Claude Sonnet 5·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Kimi K2 Thinking·solo · GPT-5.1 Chat (latest)·solo · GPT-5.1·low · GPT-5.1·medium · GPT-5.1·high · Gemini 3 Pro·high · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · GLM-4.7·think · Gemini 3 Flash·minimal · Gemini 3 Flash·high · Gemini 3 Flash·medium · Gemini 3.1 Pro·low · Gemini 3.1 Pro·medium · Gemini 3.1 Pro·high · Qwen3.5-122B-A10B·nothink · GPT-5.2·low · GPT-5.2·nothink · GPT-5.2·medium · GPT-5.2·high · GPT-5.2 Chat (latest)·solo · GPT-5.4·nothink · GPT-5.4·low · GPT-5.4·medium · GPT-5.4·high · GPT-5.4·xhigh · Qwen3.5-35B-A3B·nothink · Qwen3.5-27B·nothink · GPT-5.5·nothink · GPT-5.5·low · GPT-5.5·medium · GPT-5.5·high · GPT-5.5·xhigh · Grok 4.20·nothink · Grok 4.20·think · Qwen3.6-27B·nothink · Gemini 3.5 Flash·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash·high · Grok 4.5·high · GPT-5.6 Luna·nothink · GPT-5.6 Luna·low · GPT-5.6 Luna·medium · GPT-5.6 Luna·high · GPT-5.6 Luna·xhigh · GPT-5.6 Terra·nothink · GPT-5.6 Terra·low · GPT-5.6 Terra·medium · GPT-5.6 Terra·high · GPT-5.6 Terra·xhigh · GPT-5.6 Sol·nothink · GPT-5.6 Sol·low · GPT-5.6 Sol·medium · GPT-5.6 Sol·high · GPT-5.6 Sol·xhigh · Grok 4.6·high · Gemini 3.6 Flash·minimal · Gemini 3.6 Flash·medium · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·medium · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·medium · Gemini 3.8 Flash·high · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · Grok 4.7·high · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh |
| Vals AIME vals.ai | Vals AI · AIME, compute_effort null (effort unverified), board 2026-04-16 ✓ | xmodel | Claude Opus 4.7·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·think · Kimi K2 Thinking·solo · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high |
| Vals BioMystery vals.ai | Vals AI · BioMysteryBench (Terminus), Sonnet 5.5, Opus 5.5 and Opus 5 at max — snapshot 27 Sep, re-priced ✓ | xmodel | GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max |
| Vals CaseLaw v2 vals.ai | Vals AI · CaseLaw v2, compute_effort null (effort unverified), board 2026-05-04 ✓ | xmodel | Claude Opus 4.7·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Grok 4.1 Fast·think · Grok 4.20·think · Grok 4.3·high · Kimi K2 Thinking·solo · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh |
| Vals Code Migration vals.ai | Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Fable 5·max · Claude Fable 5.1·max · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Opus 5.5·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K2.7 Code·solo · Kimi K3·max · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals CorpFin v2 vals.ai | Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Opus 5·max · Qwen3.8 Max·xhigh · DeepSeek V4 Flash 0731 build·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·xhigh · GLM-5.2·max |
| Vals CUA-bench vals.ai | Vals AI · CUA-bench via Claude Code, Opus 5.5, Opus 5 and Fable 5.1 at max — snapshot 27 Sep, re-priced (board 22 Sep) ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5·max · Claude Opus 5.5·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · GPT-5.6 Sol·max · GPT-6 Astra·max |
| Vals CyberBench vals.ai | Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5·max · Claude Fable 5.1·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Opus 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals Finance Agent v2 vals.ai | Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·high · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.2·max |
| Vals GPQA vals.ai | Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Opus 5·max · Qwen3.8-27B·xhigh · Qwen3.8 Max·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·xhigh · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals HLAB vals.ai | Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Fable 5·max · Claude Fable 5.1·max · Claude Opus 4.7·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Opus 5.5·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals Index (29 Sep) vals.ai | vals.ai Vals Index as recomputed on 29 Sep 2026 (older models dropped), max effort, accuracy and cost per test ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 5·max · Claude Fable 5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Haiku 5.5·max · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.4 mini·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max |
| Vals IOI vals.ai | Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3.1 Pro·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.3-Codex·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals Legal Research vals.ai | Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max · Claude Haiku 5.5·max · Claude Fable 5.2·high · Gemini 4 Carbon·high |
| Vals LegalBench vals.ai | Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals LiveCodeBench vals.ai | Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Qwen3.8-27B·xhigh · Qwen3.8 Max·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K2.7 Code·solo · Kimi K3·max · GPT-5.1·high · GPT-5.1-Codex-Max·high · GPT-5.2·xhigh · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Sol·max · GPT-5.6 Terra·xhigh · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals MedCode vals.ai | Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max |
| Vals MedQA vals.ai | Vals AI · MedQA, Sonnet 4.6 vs Haiku 4.5 thinking, effort null, board 2026-04-16 ✓ | xmodel | Gemini 3 Flash·high · Gemini 3.1 Pro·high · Grok 4.1 Fast·nothink · Grok 4.1 Fast·think · Grok 4.20·think · Kimi K2 Thinking·solo · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh |
| Vals MedScribe vals.ai | Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals MMLU-Pro vals.ai | Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Qwen3.8-27B·xhigh · Qwen3.8 Max·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·xhigh · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals MMMU vals.ai | Vals AI · MMMU, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Qwen3.8-27B·xhigh · Qwen3.8 Max·xhigh · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GLM-5.3-Flash·max |
| Vals MortgageTax vals.ai | Vals AI · MortgageTax, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Opus 5·max · Qwen3.8-27B·xhigh · Qwen3.8 Max·xhigh · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max |
| Vals MysteryMechanism vals.ai | Vals AI · MysteryMechanism, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·high · Grok 4.7·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max |
| Vals ProgramBench vals.ai | Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Fable 5·max · Claude Fable 5.1·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Opus 5.5·max · Claude Sonnet 5·max · Claude Haiku 5.5·max · Claude Sonnet 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.3·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K2.7 Code·solo · Kimi K3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals ProofBench vals.ai | Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓ | xmodel | GLM-5.3·max · GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max · Claude Fable 5·max · Claude Fable 5.1·max · Claude Opus 5·max · Claude Opus 5.5·max · Claude Sonnet 5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.6 Luna·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6.1 Sol·max · GLM-5.3-Flash·max |
| Vals Public Benefits v1 vals.ai | Vals AI · Public Benefits Bench v1 (board 2026-06-09), max + Haiku default ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Grok 4.3·high · GPT-5.5·xhigh |
| Vals Public Benefits v1.1 vals.ai | Vals AI · Public Benefits v1.1, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·max · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.3·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max |
| Vals SAGE vals.ai | Vals AI · grading student math work against a rubric, all at max ✓ | xmodel | GPT-6 Luna·max · GPT-6 Sol·max · Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Qwen3.8-27B·xhigh · DeepSeek V4.1 Flash·high · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6.1 Sol·max |
| Vals Science Index (biology data investigation, mechanism discovery, research workflows) vals.ai | vals.ai Science Index, equal-weighted BioMysteryBench / MysteryMechanism / Terminal-Bench-Science; cost per test; read 2026-10-18 ✓ | xmodel | Claude Opus 5·max · DeepSeek V4.1 Flash·high · Gemini 3.1 Pro·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.6·high · Grok 4.7·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max |
| Vals SkillsBench vals.ai | Vals AI · SkillsBench with skills, OpenHands, 5 Claude models at max — snapshot 27 Sep, re-priced ✓ | xmodel | GLM-5.3·max · GPT-5.6 Sol·max |
| Vals SRE-Bench vals.ai | Vals AI · binary reverse engineering (despite the name), all at max — snapshot 27 Sep, re-priced ✓ | xmodel | Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · Claude Sonnet 5·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.5·high · Grok 4.7·xhigh · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals SWE-bench vals.ai | Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Opus 5·max · Qwen3.8-27B·xhigh · Qwen3.8 Max·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K2.7 Code·solo · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.3-Codex·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GLM-5.2·max · GLM-5.3·max · GLM-5.3-Flash·max |
| Vals Tax Agent vals.ai | Vals AI · Tax Agent, 10 Claude entries at their stated compute_effort — snapshot 27 Sep, re-priced ✓ | xmodel | DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · GLM-5.3·max · GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max · Claude Opus 5.5·max · Claude Opus 5·max · Claude Fable 5.1·max · Claude Sonnet 5·max · Claude Sonnet 5.5·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Opus 4.7·max · Claude Haiku 5.5·max · Claude Fable 5.2·high |
| Vals TaxEval v2 vals.ai | Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | GLM-5.3·max · GPT-5.6 Sol·max · Claude Fable 5.1·max · Claude Fable 5·max · Claude Opus 4.8·max · Claude Sonnet 5·max · Claude Opus 4.7·max · Claude Opus 5·max · Qwen3.8-27B·xhigh · Qwen3.8 Max·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.7 Flash·high · Gemini 3.8 Flash·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Kimi K3·max · GPT-5.1·high · GPT-5.2·xhigh · GPT-5.4·xhigh · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Terra·xhigh · GLM-5.2·max · GLM-5.3-Flash·max |
| Vals TB 2.1 vals.ai | Vals AI · Terminal-Bench 2.1, Claude models at their stated compute_effort (Sonnet 5.5, Opus 5.5 high) — snapshot 27 Sep, re-priced ✓ | xmodel | GLM-5.2·max · GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max · Claude Opus 4.8·max · Kimi K2.7 Code·solo · GPT-5.5·xhigh |
| Vals TB 4.0 vals.ai | Vals AI · Terminal-Bench 4.0, all Claude at max, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · GLM-5.3·max · GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max |
| Vals TB-Science vals.ai | Vals AI · Terminal-Bench-Science, all at max — snapshot 27 Sep, re-priced ✓ | xmodel | GLM-5.3·max · GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max |
| Vals Time Horizon KSP vals.ai | Vals AI · Time Horizon Index: KSP, 3 Claude at max, board 2026-09-14 ✓ | xmodel | Claude Fable 5.1·max · Claude Opus 4.8·max · Claude Opus 5·max · Claude Opus 5.5·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.5·high · Grok 4.6·high · Kimi K3·max · GPT-5.5·xhigh · GPT-5.6 Sol·max · GPT-6 Astra·max |
| Vals Vibe Code Bench 1-100 vals.ai | Vals AI · Vibe Code Bench 1-100, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Gemini 4 Carbon·high · GLM-5.3·max · GPT-5.6 Sol·max |
| vals.ai Robo-bench (robot arm + humanoid control in a physics sim, most tasks held out) vals.ai | claude-code harness; overall score; cost = vals 'Est. API cost' per test (USD, estimate); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · Gemini 4 Argon·high · Grok 4.7·xhigh · GPT-6 Astra·max · GPT-6.1 Sol·max |
| vals.ai SWE-bench Multimodal (JS front-end issues with screenshots) vals.ai | Vals-run agent; resolve rate overall, cost per test (USD); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓ | xmodel | Claude Opus 5.5·max · Claude Fable 5.1·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · DeepSeek V4.1 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max |
| valscodemig11 vals.ai | — | xmodel | Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max · Claude Fable 5.2·high |
| valsemb11 vals.ai | — | xmodel | Claude Fable 5.1·max · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max · Claude Fable 5.2·high · Gemini 4 Carbon·high |
| valsfab21 vals.ai | — | xmodel | Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · Claude Fable 5.2·high · Gemini 4 Carbon·high |
| valsfincoach vals.ai | — | xmodel | GLM-5.3·max · GPT-5.6 Sol·max · GPT-6 Luna·max · GPT-6 Sol·max |
| valsfinindex vals.ai | — | xmodel | Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · Claude Fable 5.2·high · Gemini 4 Carbon·high |
| valshlab-2026-10-21 vals.ai | — | xmodel | Claude Opus 5.5·max · Claude Sonnet 5.5·max · Claude Haiku 5.5·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max · Gemini 4 Carbon·high |
| valsimmigration vals.ai | — | xmodel | GLM-5.3·max · GPT-6 Luna·max |
| valsindex4 vals.ai | — | xmodel | DeepSeek V4.1 Flash 1017 build·max · Gemini 4 Carbon·high · GLM-5.3·max · GPT-6 Luna·max · Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4 Pro 0813 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6.1 Sol·max · Claude Fable 5.2·high |
| valsinfprovider vals.ai | — | xmodel | DeepSeek V4.1 Flash·high · Kimi K3·max |
| valslegalindex vals.ai | — | xmodel | Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max · Claude Fable 5.2·high · Gemini 4 Carbon·high |
| valssage11 vals.ai | — | xmodel | Claude Haiku 5.5·max · Claude Opus 5.5·max · Claude Sonnet 5.5·max · DeepSeek V4.1 Flash·max · Gemini 3.5 Flash-Lite·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.6 Terra·max · GPT-6 Luna·max · GPT-6.1 Sol·max · GLM-5.3-Flash·max |
| valstaxwp vals.ai | — | xmodel | Qwen3.8-27B·xhigh · DeepSeek V4 Flash 0731 build·high · DeepSeek V4 Pro 0813 build·max · DeepSeek V4.1 Flash·max · DeepSeek V4.1 Flash 1017 build·max · Gemini 3 Flash·high · Gemini 3.1 Pro·high · Gemini 3.5 Flash-Lite·high · Gemini 3.6 Flash·high · Gemini 3.8 Flash·high · Gemini 4 Argon·high · Grok 4.20·think · Grok 4.3·high · Grok 4.5·high · Grok 4.6·high · Grok 4.7·xhigh · Kimi K3·max · GPT-5.4 mini·xhigh · GPT-5.4 nano·high · GPT-5.5·xhigh · GPT-5.6 Luna·max · GPT-5.6 Sol·max · GPT-5.6 Terra·max · GPT-6 Astra·max · GPT-6 Luna·max · GPT-6 Sol·max · GPT-6.1 Sol·max · GLM-5.3·max · GLM-5.3-Flash·max · Claude Haiku 5.5·max · Claude Fable 5.2·high · Gemini 4 Carbon·high |
| valswsindex vals.ai | — | xmodel | Claude Fable 5·max · Grok 4.5·high · GPT-5.6 Sol·max |
| Vectara HHEM-2.3 summarisation faithfulness (own 7700-article set) vectara-hhem | Vectara hallucination leaderboard README 'Last updated on October 16, 2026'; each LLM summarises every document with the stated prompt (summary <=20% of length); judge HHEM-2.3; answer rate and avg length given; no tools; score = Factual Consistency Rate (100 - hallucination rate); live snapshot; read 2026-10-19 ✓ | xmodel | GPT-5.2·low · GPT-5.2·high · GPT-5.1·low · GPT-5.1·high |
| Vectorise doc-search, judge-graded (Vidali gist) vidali-gist | citation recall by cross-model judge, 22 Sep 2026 evening run ✓ | xmodel | Claude Opus 5·high · Claude Sonnet 5·high |
| Vending-Bench 2 (Andon Labs) andonlabs-vb2 | andonlabs.com/evals/vending-bench-2: mean final money balance after a simulated year (start $500), mean of runs; 3 rows read from the page, the rest relayed by Epoch's mirror; no effort stated by the page; no cost; read 2026-10-15 ✓ | xmodel | Claude Opus 4.8·high · Claude Fable 5·high · Claude Fable 5·low · Claude Fable 5·max · Claude Fable 5·medium · Claude Opus 4.8·max · Kimi K2.7 Code·solo |
| VeriContest-Lean (theorems proved of 10; cost per proved theorem) arxiv-2610.03994 | 10 theorems x 5 Claude models x 5 effort settings, one attempt each, Claude Code, 10 iterations/3 h; Table 4-5 of arXiv 2610.03994; read 2026-10-15 ✓ | xmodel | Claude Fable 5.1·low · Claude Fable 5.1·medium · Claude Fable 5.1·high · Claude Fable 5.1·xhigh · Claude Fable 5.1·max · Claude Opus 5.5·low · Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh · Claude Opus 5.5·max · Claude Sonnet 5.5·low · Claude Sonnet 5.5·medium · Claude Sonnet 5.5·high · Claude Sonnet 5.5·xhigh · Claude Sonnet 5.5·max · Claude Haiku 5.5·low · Claude Haiku 5.5·medium · Claude Haiku 5.5·high · Claude Haiku 5.5·xhigh · Claude Haiku 5.5·max · GPT-6 Astra·low · GPT-6 Astra·medium · GPT-6 Astra·high · GPT-6 Astra·xhigh · GPT-6 Astra·max · GPT-6.1 Sol·low · GPT-6.1 Sol·medium · GPT-6.1 Sol·high · GPT-6.1 Sol·xhigh · GPT-6.1 Sol·max · GPT-6 Sol·low · GPT-6 Sol·medium · GPT-6 Sol·high · GPT-6 Sol·xhigh · GPT-6 Sol·max · GPT-6 Luna·high · GPT-6 Luna·xhigh · GPT-6 Luna·max |
| VEX-Bench arxiv-2609.08040 | Claude Code 2.1.150 · high, Opus 4.6 vs Sonnet 4.6, cost+tokens per case ✓ | xmodel | GPT-5.5·xhigh · GPT-5.4 mini·xhigh |
| vibe-openscad iter-png-1 github-vibeopenscad | vibe-openscad · 7 tasks, 1 PNG-feedback round, 5 Claude models ✓ | xmodel | Claude Haiku 4.5·nothink · Claude Opus 4.8·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Flash-Lite·minimal |
| vibe-openscad iter-png-2 github-vibeopenscad | vibe-openscad · 7 tasks, 2 PNG rounds ✓ | xmodel | Claude Haiku 4.5·nothink · Claude Opus 4.8·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Flash-Lite·minimal |
| vibe-openscad iter-png-3 github-vibeopenscad | vibe-openscad · 7 tasks, 3 PNG rounds ✓ | xmodel | Claude Haiku 4.5·nothink · Claude Opus 4.8·nothink · Claude Sonnet 4.6·nothink · Gemini 3.1 Flash-Lite·minimal |
| vibe-openscad PDF page github-vibeopenscad | vibe-openscad · 2 tier-4 datasheet tasks, 8 Claude models default ✓ | xmodel | Claude Fable 5·high · Claude Fable 5.1·high · Claude Haiku 4.5·nothink · Claude Opus 4.7·nothink · Claude Opus 4.8·nothink · Claude Opus 5·high · Claude Sonnet 4.6·nothink · Claude Sonnet 5·high · Gemini 3.1 Flash-Lite·minimal · Gemini 3.5 Flash·medium · Gemini 3.5 Flash-Lite·minimal · GPT-5.4·nothink · GPT-5.5·medium · GPT-5.6 Luna·medium · GPT-5.6 Terra·medium |
| vibetraining-avg arxiv-2610.26342 | — | xmodel | GPT-6 Astra·xhigh · GPT-5.6 Sol·xhigh |
| Video2World (222 instances), coding agents arxiv-2610.04432 | Video2World (arXiv 2610.04432, 3 Oct 2026): 222 instances (189 clips, 39 task families), agent builds a simulation package from video; score = V2WScore (0-100, first column of main table; build rate, functionality success, progress, geometry errors are other columns); seven models in OpenCode (via OpenRouter), GPT-6 Astra in Codex CLI, Fable-5.1 in Claude Code; cost = Table A17 mean API cost per episode (USD); read 2026-10-20 ✓ | xmodel | Claude Fable 5.1·max · Gemini 3.8 Flash·high |
| vlm-exam counting github-vlmexam | roboflow/vlm-exam · 74 samples, LLM-judge accuracy ✓ | xmodel | Claude Fable 5·low · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Sonnet 5·low · Claude Opus 4.8·low · Claude Opus 5·low · Claude Opus 5.5·low · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · GPT-5.5·low · GPT-5.5·high · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.6 Sol·low · GPT-5.6 Sol·high · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Sol·low · GPT-6 Sol·high · GPT-6 Luna·low · GPT-6 Luna·high · GPT-6.1 Sol·low · GPT-6.1 Sol·high · Qwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink · GLM-5.3-Flash·low · Kimi K2.6·nothink · Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high · Gemini 4 Argon·low · Gemini 4 Argon·high |
| vlm-exam detection github-vlmexam | roboflow/vlm-exam · 250 images, mAP@50 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Sonnet 5·low · Claude Opus 4.8·low · Claude Opus 5·low · Claude Opus 5.5·low · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · GPT-5.5·low · GPT-5.5·high · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.6 Sol·low · GPT-5.6 Sol·high · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Sol·low · GPT-6 Sol·high · GPT-6 Luna·low · GPT-6 Luna·high · GPT-6.1 Sol·low · GPT-6.1 Sol·high · Qwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink · GLM-5.3-Flash·low · Kimi K2.6·nothink · Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high · Gemini 4 Argon·low · Gemini 4 Argon·high |
| vlm-exam extraction github-vlmexam | roboflow/vlm-exam · 97 samples, LLM-judge accuracy; same models as OCR ✓ | xmodel | Claude Fable 5·low · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Sonnet 5·low · Claude Opus 4.8·low · Claude Opus 5·low · Claude Opus 5.5·low · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · GPT-5.5·low · GPT-5.5·high · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.6 Sol·low · GPT-5.6 Sol·high · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Sol·low · GPT-6 Sol·high · GPT-6 Luna·low · GPT-6 Luna·high · GPT-6.1 Sol·low · GPT-6.1 Sol·high · Qwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink · GLM-5.3-Flash·low · Kimi K2.6·nothink · Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high · Gemini 4 Argon·low · Gemini 4 Argon·high |
| vlm-exam identification github-vlmexam | roboflow/vlm-exam · 32 samples, near ceiling ✓ | xmodel | Claude Fable 5·low · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Sonnet 5·low · Claude Opus 4.8·low · Claude Opus 5·low · Claude Opus 5.5·low · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · GPT-5.5·low · GPT-5.5·high · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.6 Sol·low · GPT-5.6 Sol·high · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Sol·low · GPT-6 Sol·high · GPT-6 Luna·low · GPT-6 Luna·high · GPT-6.1 Sol·low · GPT-6.1 Sol·high · Qwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink · GLM-5.3-Flash·low · Kimi K2.6·nothink · Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high · Gemini 4 Argon·low · Gemini 4 Argon·high |
| vlm-exam OCR github-vlmexam | roboflow/vlm-exam · 37 samples, similarity; Fable 5.1 & Opus 5.5 low/high ×3 runs, others low ×1 ✓ | xmodel | Claude Fable 5·low · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Sonnet 5·low · Claude Opus 4.8·low · Claude Opus 5·low · Claude Opus 5.5·low · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3 Flash·low · Gemini 3.1 Pro·low · GPT-5.5·low · GPT-5.5·high · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.6 Sol·low · GPT-5.6 Sol·high · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Sol·low · GPT-6 Sol·high · GPT-6 Luna·low · GPT-6 Luna·high · GPT-6.1 Sol·low · GPT-6.1 Sol·high · Qwen3.8 Max·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink · GLM-5.3-Flash·low · Kimi K2.6·nothink · Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high · Gemini 4 Argon·low · Gemini 4 Argon·high |
| vlmexam-text github-vlmexam | — | xmodel | Claude Fable 5·low · Claude Fable 5·high · Claude Fable 5.1·low · Claude Fable 5.1·high · Claude Sonnet 5·low · Claude Sonnet 5·high · Claude Opus 4.8·low · Claude Opus 4.8·high · Claude Opus 5·low · Claude Opus 5·high · Claude Opus 5.5·low · Claude Opus 5.5·high · Claude Sonnet 5.5·low · Claude Sonnet 5.5·high · Claude Haiku 5.5·low · Claude Haiku 5.5·high · Gemini 3.5 Flash·high · Gemini 3.6 Flash·low · Gemini 3.6 Flash·high · Gemini 3.7 Flash·low · Gemini 3.7 Flash·high · Gemini 3.8 Flash·low · Gemini 3.8 Flash·high · Gemini 3.5 Flash-Lite·low · Gemini 3.5 Flash-Lite·high · Gemini 3 Flash·low · Gemini 3 Flash·high · Gemini 3.1 Pro·low · Gemini 3.1 Pro·high · GPT-5.5·low · GPT-5.5·high · GPT-5.4 mini·low · GPT-5.4 mini·high · GPT-5.6 Sol·low · GPT-5.6 Sol·high · GPT-5.6 Terra·low · GPT-5.6 Terra·high · GPT-5.6 Luna·low · GPT-5.6 Luna·high · GPT-6 Astra·low · GPT-6 Astra·high · GPT-6 Sol·low · GPT-6 Sol·high · GPT-6 Luna·low · GPT-6 Luna·high · GPT-6.1 Sol·low · GPT-6.1 Sol·high · Qwen3.8 Max 0902 build·low · Qwen3.7 Flash·nothink · Qwen3.8 Flash·nothink · Qwen3.5-27B·nothink · Qwen3.8-27B·low · Qwen3.6-27B·nothink · GLM-5.3-Flash·low · GLM-5.3-Flash·high · Kimi K2.6·nothink · Kimi K3·high · Grok 4.5·low · Grok 4.5·high · Grok 4.6·low · Grok 4.6·high · Grok 4.7·low · Grok 4.7·high · DeepSeek V4 Flash 0731 build·high · Gemini 4 Argon·low · Gemini 4 Argon·high |
| vlnr2r arxiv-2607.26148 | — | xmodel | Claude Opus 5·high · GPT-5.5·medium · GPT-5.6 Sol·low · Claude Sonnet 5·max · Claude Opus 4.8·max · Claude Fable 5·max · Claude Opus 5·max · GPT-5.5·xhigh · GPT-5.6 Sol·xhigh |
| vulcanbench3 github-vulcanbench | — | xmodel | Grok 4.5·medium · Grok 4.5·high · Claude Fable 5·low · GPT-5.6 Sol·high · Claude Fable 5·high · Grok 4.5·low · GPT-5.6 Sol·medium · Claude Fable 5·medium · GPT-5.6 Sol·low · Kimi K3·max |
| WeirdML v3 (agentic, token-axis area score) epoch-archive | 11 ML tasks x 15 configurations; score = token-axis area 0-1 with 95% CI; cost = mean API $/run; agent claude_code; read 2026-10-01 ✓ | xmodel | GPT-6 Astra·xhigh · GPT-6.1 Sol·xhigh · Claude Opus 5.5·xhigh · Claude Fable 5.1·xhigh · Claude Sonnet 5.5·xhigh · GPT-6 Sol·xhigh · Claude Opus 5·xhigh · Claude Fable 5·xhigh · GPT-5.6 Sol·xhigh · GPT-6 Luna·xhigh · Grok 4.7·xhigh · Gemini 3.8 Flash·high · DeepSeek V4.1 Flash·high · GPT-5.6 Luna·xhigh |
| WhiteKUMALabo F5/O4.8 T2 whitekumalabo | Claude Code 2.1.170 · intro-text, blind 15pt judge ✓ | xmodel | Claude Fable 5·high · Claude Opus 4.8·high |
| WhiteKUMALabo F5/O4.8 T3 whitekumalabo | Claude Code 2.1.170 · task decomposition, blind 15pt judge ✓ | xmodel | Claude Fable 5·high · Claude Opus 4.8·high |
| WhiteKUMALabo F5/O4.8 T4 whitekumalabo | Claude Code 2.1.170 · 1-sentence summary, cost/speed only ✓ | xmodel | Claude Fable 5·high · Claude Opus 4.8·high |
| wmedia-php-hidden-cases wmedia.es | — | xmodel | Claude Opus 5.5·medium · Claude Opus 5.5·high · Claude Opus 5.5·xhigh |
| yama3133 · 400k-character log qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, 400k-character log, needle arithmetic; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Claude Opus 5.5·medium · Claude Opus 5·medium · Claude Sonnet 5·medium |
| yama3133 · 8 closed-answer problems × 3 qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, 8 closed-answer problems × 3; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Claude Opus 5.5·medium · Claude Opus 5·medium · Claude Sonnet 5·medium |
| yama3133 · agentic bug fix qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, agentic bug fix, 5 planted bugs, 21 tests; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Claude Opus 5.5·medium · Claude Opus 5·medium · Claude Sonnet 5·medium |
| youkiti-prisma-tsuge10-accuracy github-youkiti | — | xmodel | Grok 4.6·low · Gemini 3.8 Flash·medium · GPT-5.6 Terra·nothink · Gemini 3 Flash·low · Gemini 3 Pro·high · GPT-5.6 Sol·nothink · GPT-5.6 Luna·nothink · Claude Opus 4.7·low · GPT-6 Astra·high |
| zapierab-domain-finance zapier | — | xmodel | Gemini 4 Carbon·medium · Claude Fable 5.2·max |
| zapierab-domain-hr zapier | — | xmodel | Gemini 4 Argon·high · Claude Fable 5.2·max |
| zapierab-domain-marketing zapier | — | xmodel | Gemini 4 Carbon·medium · Claude Fable 5.2·xhigh |
| zapierab-domain-operations zapier | — | xmodel | Gemini 4 Carbon·high · Gemini 4 Argon·medium |
| zapierab-domain-sales zapier | — | xmodel | Gemini 4 Argon·medium · Gemini 4 Carbon·high |
| zapierab-domain-support zapier | — | xmodel | Gemini 4 Carbon·medium · Gemini 4 Argon·high |
| zcarbon-domain-finance zapier | — | xmodel | Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high |
| zcarbon-domain-hr zapier | — | xmodel | Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high |
| zcarbon-domain-marketing zapier | — | xmodel | Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high |
| zcarbon-domain-operations zapier | — | xmodel | Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high |
| zcarbon-domain-sales zapier | — | xmodel | Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high |
| zcarbon-domain-support zapier | — | xmodel | Gemini 4 Carbon·low · Gemini 4 Carbon·medium · Gemini 4 Carbon·high |
| zhenhuanbench-reasoning arxiv-2610.26059 | — | xmodel | Qwen3.8-27B·low · Qwen3.8-27B·medium · Qwen3.8-27B·xhigh |
| τ^τ-Bench arxiv-2609.04611 | arXiv 2609.04611 · 53 agent-construction tasks in Claude Code, Opus 5 vs Sonnet 5 both at max, build spend at list ✓ | xmodel | Claude Opus 5·max · Claude Sonnet 5·max · GPT-5.6 Sol·xhigh · GPT-5.6 Terra·xhigh · Kimi K3·max |
How the numbers are built — the method
Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.
In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.
The procedure
- Every number traced to its source. Each source is recorded as facts: the value as the source writes it, the exact text or table cell it is read from, the URL and the date of the capture; agents read every value line by line, and a measurement whose facts lack their proof stays out. Registries of models (the names each was served under, its effort settings and defaults, dated), routes (makers' APIs, resellers, routers, harnesses), benchmarks and their versions, publishers and dated prices let one build decide, by a fixed decision tree, which model, configuration and price each measurement belongs to.
- Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale, one version of a board; a group that mixes them is split. Every run is dated, and its model and effort are read as the maker and every route in between served them on that date — never assumed: a route the source does not name is never presumed to be the maker's API, and every route that could have served the run must give the same configuration. At the least doubt on the model, the effort or what was run, the measurement stays in the data file and out of the fit.
- Costs at today's prices. Every cost is put on the model's current price from the prices in force on its run date: the source's own figure at that price, the price change applied to the run's own token counts, or, when the token mix is unknown, a price band whose width is added to the cost's precision. A cost that cannot be brought to the current price stays out.
- Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The scale is read from the metric's label, never from the benchmark's identity; whether lower is better is read from the benchmark's registry, and such a score is turned in its own unit first (an attack success rate of 30 % reads 70 %); a score whose direction is not established stays out. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
- One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result. The sizes of these effects are estimated for each model maker on its own, not tied to the other makers'; a maker's own charts about its own models carry an estimated home effect, also each maker's own, left out of every value. The fit takes the couples of eight makers measured in the same groups (Anthropic, OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi), on one scale; this page shows the current models of every maker, and the older couples no other model of their maker beats, but no model that another model of the same maker entirely outclasses. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
- Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source; the one weight set by rule is that of a single run: when a source gives k quality measurements of one run (several metrics, sub-scores or slices of the same executions), each weighs 1/k, so the run counts as one measurement.
- Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, a board copying another's numbers, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. Versions of one board that re-grade the same runs (recognised by identical costs) count those runs once. Before the fit, two groups that still share identical costs on three couples or more stop it until the copy is resolved or the coincidence proven from the sources. An index keeps, for a couple whose index components are already in the data, only what its missing components add, or is left out.
- Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
- From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.
Key values
- Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
- Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
- Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit 74 figures on this page can move by one unit in that digit (the cost of DeepSeek V4 Pro 0813 build at high, the cost of DeepSeek V4 Pro 0813 build at low, the cost of DeepSeek V4 Pro 0813 build at max, the cost of Claude Fable 5.1 at high, the cost of Claude Fable 5.1 at low, the cost of Claude Fable 5.1 at max, the cost of Claude Fable 5.1 at medium, the cost of Claude Fable 5.1 at xhigh, the cost of Claude Fable 5.2 at high, the cost of Claude Fable 5.2 at max, the cost of Claude Fable 5.2 at medium, the cost of Claude Fable 5.2 at xhigh, the cost of Claude Fable 5 at high, the cost of Claude Fable 5 at low, the cost of Claude Fable 5 at max, the cost of Claude Fable 5 at medium, the cost of Claude Fable 5 at xhigh, the cost of Gemini 3.8 Flash at high, the cost of Gemini 4 Argon at high, the cost of Gemini 4 Carbon at high, the cost of Gemini 4 Carbon at low, the cost of Gemini 4 Carbon at medium, the cost of Gemini 4 Hydrogen at xhigh, the cost of GLM-5.1 at think, the cost of GLM-5.2 at max, the cost of GPT-5.1-Codex-Max at high, the cost of GPT-5.2 Pro at high, the cost of GPT-5.2 Pro at medium, the cost of GPT-5.2 Pro at xhigh, the cost of GPT-5.2 at xhigh, the cost of GPT-5.4 Pro at xhigh, the cost of GPT-5.4 at high, the cost of GPT-5.4 at xhigh, the cost of GPT-5.5 Pro at xhigh, the cost of GPT-5.5 at high, the cost of GPT-5.5 at medium, the cost of GPT-5.5 at xhigh, the cost of GPT-5.6 Sol at max, the cost of GPT-5.6 Sol at xhigh, the cost of GPT-5.6 Terra at max, the cost of GPT-6 Astra at high, the cost of GPT-6 Astra at max, the cost of GPT-6 Astra at medium, the cost of GPT-6 Astra at xhigh, the cost of Grok 4.7 at high, the cost of Grok 4.7 at medium, the cost of Grok 4.7 at xhigh, the cost of Kimi K3 at high, the cost of Kimi K3 at max, the cost of Claude Opus 4.7 at high, the cost of Claude Opus 4.7 at max, the cost of Claude Opus 4.7 at medium, the cost of Claude Opus 4.7 at nothink, the cost of Claude Opus 4.7 at xhigh, the cost of Claude Opus 4.8 at high, the cost of Claude Opus 4.8 at max, the cost of Claude Opus 4.8 at medium, the cost of Claude Opus 4.8 at nothink, the cost of Claude Opus 4.8 at xhigh, the cost of Claude Opus 5.5 at max, the cost of Claude Opus 5.5 at xhigh, the cost of Claude Opus 5 at high, the cost of Claude Opus 5 at low, the cost of Claude Opus 5 at max, the cost of Claude Opus 5 at medium, the cost of Claude Opus 5 at xhigh, the cost of Qwen3-Coder-Next at nothink, the cost of Qwen3.8 Max at xhigh, the cost of Claude Sonnet 5.5 at max, the cost of Claude Sonnet 5 at max, the cost of Claude Sonnet 5 at xhigh, the quality of GPT-6 Luna Pro at medium, the quality of Claude Opus 4.7 at nothink, the quality of Qwen3.8 Max 0902 build at medium), far inside their intervals.
- Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.
Techniques and software
- Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes, six chains whose draws continue in the same run until the criteria are met (CmdStan only to resume a stopped fit from its stored draws, or to restart a chain left in a far region from a good chain's last draw); built and checked stage by stage following the Bayesian workflow.
- Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
- Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
- Scales: the empirical logit of bounded scores, with the binomial shape of its noise.
Checks (on the fit of 29 September 2026, before the other makers' couples joined the data). Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.