Quality vs. cost of every Kimi model, at each effort level
One point per couple: a model at one configuration (thinking off, minimal, low → max, thinking on); no line joins a model's couples. Only the couples of the Pareto frontier are labelled (model · effort), where there is room; hover or tap any point for its model, effort, cost and score. Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward.
Best Kimi model for the price — the Pareto frontier
The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated; the frontier couples are labelled (model · effort) where there is room — hover or tap any point for its model, effort, cost and score. No line joins a model's couples. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.
Value — distance to the price trend
The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 4 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.97, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.
| Model · effort | cost | expected score | value index trend = 100 |
|---|---|---|---|
| Kimi K2.7 Code · solo | 2.9× | 54.9 % | 114 |
| Kimi K3 · Max | 9.6× | 62.1 % | 99 |
| Kimi K3 · High | 8.6× | 60.2 % | 84 |
| Kimi K3 · Low | 2.6× | 51.8 % | 81 |
| Kimi K2.6 · Thinking off | 1.0× | 43.6 % | 55 |
Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.
Best value by task tier — what to actually run
Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.
Picked as the frontier couple that sits furthest below the price trend : it costs 1.1× less than the trend charges for its quality — or, read on the other axis, it scores 0.9 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.
Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — Kimi effort levels compared
Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.
| Model | thinking off | low | medium | high | xhigh | max | thinking on | Expected score top effort |
|---|---|---|---|---|---|---|---|---|
| Kimi K3 | — | 2.62.2–3.0 | — | 8.65.0–15 | — | 9.68.6–11 | — | 62.1 % |
| Kimi K2.7 Code | 2.9merged · 2.4–3.6 | 54.9 % | ||||||
| Kimi K2.6 | 1.00.81–1.2 | — | — | — | — | — | 3.92.8–5.5 | 53.8 % |
"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. A model measured at a single setting (solo : no effort parameter, see the measurement sources) fills one merged cell. A thinking off column appears when a model is measured with its thinking switched off, a minimal one for the lowest level some models have below low; each is a configuration of its own, read from its own measurements. n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).
Value-index matrix — every model and effort against the price trend
The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.
| Model | thinking off | low | medium | high | xhigh | max | thinking on |
|---|---|---|---|---|---|---|---|
| Kimi K3 | — | 8153–122 | — | 8447–152 | — | 9985–116 | — |
| Kimi K2.7 Code | 11488–148 | ||||||
| Kimi K2.6 | 5540–77 | — | — | — | — | — | 7450–108 |
Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.
What moves everything — four factors behind the numbers
The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.
Task type
Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.
Complexity
The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).
Generation
Each release buys the same quality for less: turn on Older models to set each family's earlier models beside its current one, on the same scales.
Cache & harness
Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.
Data Sources
Every source below measured at least two (model, effort) couples on the same task; 10115 cost and 23548 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.
| Source | Actual config (harness · effort) | Type | Linked couples (same task) |
|---|---|---|---|
| AA CritPt artificialanalysis | AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Kimi K2.6·nothink |
| AA GDP-PDF artificialanalysis | AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA GDPval v2 artificialanalysis | AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA Harvey LAB artificialanalysis | AA model payload · Harvey's Legal Agent Bench (AA's own run, separate from Vals's HLAB), score-only, 8 models × ladder + 4 at max ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA HLE artificialanalysis | AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Kimi K2.6·nothink |
| AA Index v4 artificialanalysis | AA model pages · Fable 5.1 sweep low→max + Fable 5/Opus 5 at max, per-suite $ ✓ | sweep | Kimi K2 Thinking·solo |
| AA MLCR artificialanalysis | AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓ | sweep | Kimi K2.7 Code·solo · Kimi K3·max |
| AA Omniscience artificialanalysis | AA model payload · Omniscience accuracy (not the penalised index) ✓ | sweep | Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Kimi K3·max |
| AA SciCode artificialanalysis | AA model payload · SciCode, 4 models × ladder + 3 at max ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA TB 4.0 artificialanalysis | AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA-Briefcase artificialanalysis | AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA-LCR artificialanalysis | AA model payload · long-context reasoning, input-dominated cost ✓ | sweep | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max · Kimi K2.6·nothink |
| Agents on Rails · features rubyonrails | rubyonrails.org/ai + rails/ai-evals runs · 20 tickets ×3, miniswen; Opus 5 & Fable 5.1 high→max, Opus 5.5 default ✓ | sweep | Kimi K3·max |
| ARC-AGI-1 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓ | sweep | Kimi K3·max |
| ARC-AGI-2 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓ | sweep | Kimi K3·max |
| BullshitBench v2 github-bullshitbench | petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓ | sweep | Kimi K2.6·nothink |
| CursorBench 3.2 cursorbench | Cursor 3.2 · matched effort, $ cost ✓ (vendor benchmark) | sweep | Kimi K3·high · Kimi K3·low · Kimi K3·max |
| InfluMatch Set 1 (11 queries) P@5, reason on all 50 candidates arxiv-2607.05968 | KOL search P@5 %, Kimi-K2.6 as reason scorer think vs no-think; tokens per query from Table 10 (no USD); paper 2026-07-07; read 2026-10-20 ✓ | sweep | Kimi K2.6·think · Kimi K2.6·nothink |
| InfluMatch Set 1 P@5, SimPO rerank then reason top-10 arxiv-2607.05968 | KOL search P@5 %, Kimi-K2.6 as reason scorer think vs no-think; tokens per query from Table 10 (no USD); paper 2026-07-07; read 2026-10-20 ✓ | sweep | Kimi K2.6·think · Kimi K2.6·nothink |
| InfluMatch Set 2 (31 queries) P@5, reason top-10 after SimPO rerank arxiv-2607.05968 | KOL search P@5 %, Kimi-K2.6 as reason scorer think vs no-think; tokens per query from Table 10 (no USD); paper 2026-07-07; read 2026-10-20 ✓ | sweep | Kimi K2.6·think · Kimi K2.6·nothink |
| ObviousBench 0.2 benchmarklist | inspect-ai · 144 items ×3 epochs, 3 models × full ladder, measured tokens ✓ | sweep | Kimi K2 Thinking·solo · Kimi K2.7 Code·solo |
| Playcode MacBook SVG playcode | Playcode · one-shot MacBook SVG, high/xhigh/max per model, provider-billed cost (quality ordinal, not scored) ✓ | sweep | Kimi K3·high · Kimi K3·max |
| PostTrainB. arxiv-2603.08640 | papier · medium/high ✓ | sweep | Kimi K2 Thinking·solo |
| Vals VibeCode vals.ai | Vals AI · Opus 5 sweep low→max (scores), $ cost at leaderboard setting; board entries — snapshot 27 Sep, re-priced ✓ | sweep | Kimi K2.7 Code·solo |
| vlm-exam github-vlmexam | roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓ | sweep | Kimi K2.6·nothink · Kimi K3·high |
| WeirdML epoch-archive | Epoch archive · 3 models × high/max, cost per run ✓ | sweep | Kimi K3·max · Kimi K2.7 Code·solo |
| Zapier AutomationB. benchmarklist | AutomationBench 1.0.6 · 657 held-out tasks, strict pass/fail, measured tokens × list ✓ | sweep | Kimi K2.7 Code·solo |
| deepseek-v41-techreport: deepswe deepseek | DeepSWE v1.1 (Resolved), mini-SWE harness; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness mini-SWE; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | Kimi K3·max |
| deepseek-v41-techreport: gpqa deepseek | GPQA Diamond (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | Kimi K3·max |
| deepseek-v41-techreport: mathapex deepseek | MathArena Apex (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | Kimi K3·max |
| deepseek-v41-techreport: tb21 deepseek | Terminal-Bench 2.1 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | Kimi K3·max |
| FeatherBench ed-is-ai/featherbench | OpenRouter(Anthropic) · 28-task multi-domain suite (coding/data/realworld/security/tool-use), effort=high (Haiku=solo), mean cost/tokens per task-run, consolidated results/summary.json; only Opus 5.5 rung saturated ✓ | xgen | Kimi K3·max |
| Official tau2-bench leaderboard, banking_knowledge (AllTools), Claude submissions by Sierra github-sierra-tau2-bench | banking_knowledge domain, 4 trials, user simulator gpt-5.2 (low), reasoning_effort max for all; pass^1 %; read 2026-10-18 ✓ | xgen | Kimi K3·max |
| AA AutomationB. artificialanalysis | AA evaluation page · 657 tasks partial-credit, 3 models at max ✓ | xmodel | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| AA FrontierCode 1.1 (AA-run, weighted score) artificialanalysis | Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| AA MMMU-Pro artificialanalysis | Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Kimi K2.6·think · Kimi K3·max |
| AA-AnalystAgent artificialanalysis | Artificial Analysis AA-AnalystAgent; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| aa-index411 tabbit.ai | — | xmodel | Kimi K3·max |
| aaapex artificialanalysis | — | xmodel | Kimi K3·max · Kimi K2.6·think |
| aaenterpriseops artificialanalysis | — | xmodel | Kimi K2.6·think · Kimi K2.7 Code·solo |
| aagpqa artificialanalysis | — | xmodel | Kimi K3·max · Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo |
| aaifbench artificialanalysis | — | xmodel | Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo |
| aaomni-index artificialanalysis | — | xmodel | Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo · Kimi K3·max |
| aatau-banking artificialanalysis | — | xmodel | Kimi K2.6·think · Kimi K2.7 Code·solo · Kimi K3·max |
| aatau2 artificialanalysis | — | xmodel | Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo |
| aatb21 artificialanalysis | — | xmodel | Kimi K3·max · Kimi K2.6·think · Kimi K2.7 Code·solo |
| aatbhard artificialanalysis | — | xmodel | Kimi K2.6·think · Kimi K2.6·nothink · Kimi K2.7 Code·solo |
| Agent Island social-deduction arena (Plackett-Luce skill), OpenRouter models arxiv-2605.04312 | Agent Island (arXiv 2605.04312, 5 May 2026): 999 multi-agent social-deduction games with 49 models via OpenRouter, Bayesian Plackett-Luce posterior skill (mean of lambda; prior mean 1; Table 1); reasoning effort not stated; no cost; read 2026-10-20 ✓ | xmodel | Kimi K2 Thinking·solo |
| Agents on Rails · atomic rubyonrails | rubyonrails.org/ai + rails/ai-evals runs · 21 Writebook tasks ×3, miniswen, list-price cost/run ✓ | xmodel | Kimi K3·max |
| aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high) aibenchy | aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓ | xmodel | Kimi K2.6·nothink |
| AIME 2025 (GLM-4.7 card) zhipu | AIME 2025; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| ALE-Bench epoch-archive | Epoch archive · 3 models at matched high, measured token splits ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| android-bench-2 | — | xmodel | Kimi K3·max |
| APEX-Harm mean task score under attack (all 6 categories) apexharm-2610.13623 | APEX-Harm 180 attack task instances per model (15 underlying APEX-Agents tasks x 12 attack variants), APEX-Agents harness with MCP tools, 150 steps, no timer, judge/rules per paper; metric mean task score 0-1 (utility under attack); ASR/ER safety rates not extracted; arXiv 2610.13623 Table 4 (Oct 2026); cost = Table 1 total for Gemini 3.8 Flash only; read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| apexharm-asr apexharm-2610.13623 | — | xmodel | Kimi K3·max |
| arxiv-2610.13075-longbench-v2-direct arxiv-2610.13075 | — | xmodel | Kimi K3·max |
| arxiv-2610.13075-officeqa-v1-direct arxiv-2610.13075 | — | xmodel | Kimi K3·max |
| arxiv-2610.13075-officeqa-v2-direct arxiv-2610.13075 | — | xmodel | Kimi K3·max |
| arxiv-2610.22599-direction arxiv-2610.22599 | — | xmodel | Kimi K3·max |
| arxiv-2610.22599-r2 arxiv-2610.22599 | — | xmodel | Kimi K3·max |
| arxiv-2610.22599-slope arxiv-2610.22599 | — | xmodel | Kimi K3·max |
| arxiv-2610.22599-top-idea-gain arxiv-2610.22599 | — | xmodel | Kimi K3·max |
| CL-Bench clbench.com | CL-Bench, learning from context · overall score on 0–1 with its std in the flags; Opus 4.5 with and without thinking as default/nothink; no cost; read 2026-09-30 ✓ | xmodel | Kimi K2 Thinking·solo |
| ClawProBench open dataset 102 tasks (OpenClaw) clawprobench-board | ClawProBench (suyoumo.github.io/bench), open dataset 102 tasks, OpenClaw agent runtime v3.2.6, deterministic grading, repeated trials; Final Score 0-100 (page default rank signal); cost = page cost_usd = (in + 0.5 x (cacheread + cachewrite)) x price_in + out x price_out; rows updated 2026-02..2026-07 (live board); one route kept per couple (earliest updated); read 2026-10-20 ✓ | xmodel | Kimi K2 Thinking·solo |
| cliphou.se Motion Graphics Benchmark v1.1: cost per run (Remotion videos) cliphou.se | 3 briefs x 2 runs via OpenRouter, reasoning high; cost = mean OpenRouter generation cost per run over the 6 runs (computed by us); no quality score published (human review pending); read 2026-10-15 ✓ | xmodel | Kimi K3·high |
| Conceptual Reasoning Index · DTBench conceptualreasoning.ai | Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| Conceptual Reasoning Index · LMCA conceptualreasoning.ai | Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| ctala github-ctala | Claude Code CLI · reasoning not configured, temp 0.7 ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K2 Thinking·solo |
| DataLLM Lab: 9 executed Python tasks, Qwen3.8 Max family (Prime, Max, Max-0902), pass and cost per 1,000 tasks datallmlab | 9 executed Python tasks (python3 -I subprocess, 12 s timeout), temperature 0, max_tokens 4000, one scored attempt, via OpenRouter; three runs on different days (Prime 2026-10-02, Max 2026-08-20, Max-0902 2026-09-16); cost = measured tokens x list price on the priced-on date (source-computed, 'not a billing statement'); read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| datallmlab-coding9 datallmlab | — | xmodel | Kimi K2.7 Code·solo |
| DeepSeek-V3.2 report: aime deepseek | aime; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: browsecomp-cm deepseek | BrowseComp with the context-management technique (marked *), search agent with a commercial search API; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: browsecompzh deepseek | browsecompzh; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: gpqa deepseek | gpqa; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hle deepseek | hle; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hle-search deepseek | hle-search; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hmmt25 deepseek | hmmt25; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: hmmtnov deepseek | hmmtnov; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: imo deepseek | imo; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: lcb deepseek | lcb; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 3 adds Speciale and output tokens; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: mcpmark deepseek | mcpmark; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: mcpuniverse deepseek | mcpuniverse; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: mmlupro deepseek | mmlupro; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: swemulti deepseek | swemulti; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: sweverified deepseek | sweverified; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: tau2 deepseek | tau2; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| DeepSeek-V3.2 report: tooldec deepseek | tooldec; DeepSeek-V3.2 technical report Tables 2/3/9 (arXiv 2512.02556, 2025-12-02; release 2025-12-01); temperature 1.0, context window 128K; math/HLE template; judge/n not stated; Table 9 adds non-thinking V3.2; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| deepseek-v41-techreport: ale deepseek | Agents' Last Exam (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold(ALE-CLI); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: autobench deepseek | AutomationBench v1.0.6 public set (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: babyvision deepseek | BabyVision w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: chartography deepseek | Chartography w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: cybergym deepseek | CyberGym (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: hle deepseek | HLE (Pass@1), full set; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: hle-tools deepseek | HLE with tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: nl2repo deepseek | NL2Repo-Bench (Score); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: programbench deepseek | ProgramBench (Almost@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: tb30 deepseek | Terminal-Bench 3.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: tb40 deepseek | Terminal-Bench 4.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| deepseek-v41-techreport: zerobench deepseek | ZeroBench-main w/ tools (Pass@5); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Design Arena - all categories designarena | designarena.ai/leaderboard 'allcategories': crowd human-vote Elo on AI-generated designs/UI (1,817,484 votes, 180 models); effort not in the Claude slugs (other vendors' slugs carry -max/-high) so default; no cost; read 2026-10-15 ✓ | xmodel | Kimi K2.7 Code·solo |
| Design Arena - fullstack designarena | designarena.ai/leaderboard 'fullstack' category: crowd human-vote Elo on full-stack app generation (61,252 votes, 49 models); effort not stated; no cost; read 2026-10-15 ✓ | xmodel | Kimi K2.7 Code·solo |
| Design Creativity Bench (UI generation; overall T-score, median cost per design) arxiv-2610.11539 | 80 UI design tasks / 160 prompts, one generation per design, highest effort (Opus 5.5 at xhigh); Tables 2 and 9 of arXiv 2610.11539; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| dubesor-benchtable dubesor | — | xmodel | Kimi K2 Thinking·solo |
| E-Bench (MCP tools): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench (MCP tools): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| E-Bench-Code (code execution): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench-Code (code execution): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| econ-eval leaderboard A (20 tasks, Opus-5 judge) github-deluair-econ-eval | deluair/econ-eval README at commit 77f9f3e (16 Oct 2026), leaderboard of 2026-09-10: 20 economics tasks (quantitative, reasoning, coding, writing) x 5 samples = 100 completions/model, numeric/exact/code-exec graders + rubric tasks judged by Claude Opus 5; score = mean over tasks of task means; cost = total for the 100 completions from token counts x list price (prices read from the OpenRouter catalog 2026-07-31 / vendor pages); per-track tables are the same completions (not extra groups); ten models ran 2026-08-01, deepseek-flash added 2026-09-10 same tasks/samples/judge; read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| EMB vals.ai | Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Epoch · chess puzzles epoch.ai | Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K3·max · Kimi K3·high · Kimi K3·low |
| Epoch · frontiermath tier 4 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| Epoch · frontiermath tiers 1 3 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| Epoch · furniture assembly epoch.ai | Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K3·max |
| Epoch · gpqa diamond epoch.ai | Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K3·low · Kimi K3·high · Kimi K2.7 Code·solo · Kimi K3·max |
| Epoch · mystery game puzzles epoch.ai | Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K3·low · Kimi K2.6·nothink · Kimi K3·max |
| Epoch · otis mock aime 2024 2025 epoch.ai | Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K3·low · Kimi K3·high · Kimi K2.7 Code·solo · Kimi K3·max |
| Epoch · simpleqa verified epoch.ai | Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| epoch-ext-surface-evolver-bench epoch.ai | — | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| EQ-Bench creative-writing-longform (EQ-bench/eqbench-leaderboard-results) EQ-bench/eqbench-leaderboard-results (GitHub) | GitHub repo EQ-bench/eqbench-leaderboard-results, pinned commit 266853f2ac7d10c31430cb3bd00d64c758beb581, dir creative-writing-longform/results_sonnet-4.6/, 12/12 multi-chapter longform tasks, judge claude-sonnet-4-6 (flag claude-judge), 1 iteration (n=1); distinct board from the already-held eqbench4 (eqbench.com static roleplay-Elo data, different judge/metric/source) — not a restatement.; read 2026-10-08 ✓ | xmodel | Kimi K2 Thinking·solo |
| fcode20pass cognition | — | xmodel | Kimi K3·max |
| fcode20score cognition | — | xmodel | Kimi K3·max |
| Financial Touchstone v1.2 accuracy (annual-report QA, shared top-5 retrieval) arxiv-2608.08634 | arXiv 2608.08634 (9 Aug 2026) Financial Touchstone v1.2: 2,967 question-context-answer items on 495 annual reports, same top-5 retrieved context for every model, output cap 5,000 tokens, LLM judge Claude Opus 4.6 (16 models) / Claude Opus 4.8 (Kimi K2.6, GLM-4.7, Qwen3-Max, ERNIE 5.1); score = accuracy excluding retriever errors (Fig. 1); hallucination rate = same runs, other metric (not a group); read 2026-10-20 ✓ | xmodel | Kimi K2.6·think |
| FrontierSWE v2 frontierswe | Proximal frontierswe.com + Sonnet 5.5 card · 34 ultra-long tasks, max effort, proximus harness, mean reward@5; cost basis undocumented ✓ | xmodel | Kimi K3·max |
| gdpx-overall semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-arithmetic semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-beautiful-machines semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-emulated semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-ergodex semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-extrapolate semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-latchbio semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-third-moment semianalysis | — | xmodel | Kimi K3·max |
| gdpx-p-valenceforge semianalysis | — | xmodel | Kimi K3·max |
| Gert Labs GBENCH agentic coding (models write code run as agents) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode agentic_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Kimi K2.7 Code·solo |
| Gert Labs GBENCH decision making (models play the games) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode decision_making, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Kimi K2.7 Code·solo |
| Gert Labs GBENCH one-shot coding (models write code in one shot) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode oneshot_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Kimi K2.7 Code·solo |
| GPQA-Diamond (GLM-4.7 card) zhipu | GPQA-Diamond; GLM-4.7 card 2026-01-29; settings: default settings temp 1.0 top-p 0.95 131072; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| hehee9 Maze Bench Tier 1 (30 mazes) hehee9maze-bcd609c | Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 30 mazes (6 sizes x 5, 4x4..18x18), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status completed); read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| HLE (GLM-4.7 card) zhipu | HLE; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| HLE w/ tools (GLM-4.7 card) zhipu | HLE w/ tools; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| HMMT Feb 2025 (GLM-4.7 card) zhipu | HMMT Feb 2025; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| Hy-MultiTurn (Chinese deep multi-turn dialogue understanding): Importance score hymultiturn-2607.29196 | Hy-MultiTurn leaderboard (Tencent Hunyuan authors; Hy3/Doubao rows are not in the eight providers): six dimensions, final-turn response scored against weighted rubric items; Importance (weighted correct share) = primary metric, mean of three independent rolls; arXiv 2607.29196 (31 Jul 2026); read 2026-10-20 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think · Kimi K2.6·nothink |
| hypfrontier-challenge64-hf-valid hypfrontier-2608.10843 | — | xmodel | Kimi K2.7 Code·solo |
| ibmartian-full-p-other ironbee | — | xmodel | Kimi K3·low |
| InferenceBench (prompt v1.0.1 (†)) inferencebench.ai | InferenceBench, agents tuning LLM inference serving in Claude Code · geometric-mean speed-up over the PyTorch eager baseline across four scenarios, ±SEM over seed-pair runs in the flags; runs with the benchmark prompt v1.0.1 (†) only; effort from the model name; no cost; read 2026-09-30 ✓ | xmodel | Kimi K2.7 Code·solo |
| ITBench-AA (SRE) artificialanalysis | Artificial Analysis ITBench-AA (SRE); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Kimi K2.6·think · Kimi K3·max |
| K3 report Fig.13b: BrowseComp score vs cost per task moonshot | BrowseComp; K3 with context compaction at 300K tokens; GPT-5.6 Sol per effort medium/high/max (points read from the vector drawing of Fig.13b); cost USD per task as plotted (source of costs not stated); K3 tech report p.32; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| K3 report Table 3: Agent Behavior Bench moonshot | Agent Behavior Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| K3 report Table 3: Chat All-in-One Bench moonshot | Chat All-in-One Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| K3 report Table 3: DECK Bench moonshot | DECK Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| K3 report Table 3: Faithfulness moonshot | Faithfulness (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| K3 report Table 3: Finance Bench moonshot | Finance Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| K3 report Table 3: KWV Bench moonshot | KWV Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| K3 report Table 3: MIRA Bench moonshot | MIRA Bench (Moonshot in-house); harness MIRA; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| KernelBench MEGA (Claude Code) github-kernelbench | 02_kimi_linear_decode megakernel on RTX PRO 6000 Blackwell · harness=claude (Claude Code) · n=1 per cell, runs months apart (Jun-Sep 2026); NOT the board's CUDA task (that is the held kernelbench-cc/or group); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| Kimi K2.6 card: AIME 2026 moonshot | AIME 2026; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: APEX-Agents moonshot | APEX-Agents; 452 of the 480 public tasks (as Artificial Analysis); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: BabyVision moonshot | BabyVision; vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: BabyVision (w/ python) moonshot | BabyVision (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: BrowseComp (Agent Swarm) moonshot | BrowseComp (Agent Swarm); discard-all context management; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: CharXiv (RQ) moonshot | CharXiv (RQ); vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: CharXiv (RQ) (w/ python) moonshot | CharXiv (RQ) (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: Claw Eval (pass^3) moonshot | Claw Eval (pass^3); Claw Eval v1.1; max tokens per step 16,384; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: DeepSearchQA (f1-score) moonshot | DeepSearchQA (f1-score); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: HLE-Full moonshot | HLE-Full; full set; max generation 98,304 tokens; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: HLE-Full (w/ tools) moonshot | HLE-Full (w/ tools); full set; max generation 98,304 tokens; tools: search/code/browse; per-step 49,152; context management; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: HMMT 2026 (Feb) moonshot | HMMT 2026 (Feb); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: IMO-AnswerBench moonshot | IMO-AnswerBench; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: LiveCodeBench (v6) moonshot | LiveCodeBench (v6); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: MathVision moonshot | MathVision; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: MathVision (w/ python) moonshot | MathVision (w/ python); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: MCPMark moonshot | MCPMark; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: MMMU-Pro moonshot | MMMU-Pro; vision; max tokens 98,304; avg@3; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: MMMU-Pro (w/ python) moonshot | MMMU-Pro (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: OJBench (python) moonshot | OJBench (python); temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: OSWorld-Verified moonshot | OSWorld-Verified; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: SciCode moonshot | SciCode; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: SWE-Bench Multilingual moonshot | SWE-Bench Multilingual; in-house SWE-agent-derived framework; avg of 10 runs; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: Toolathlon moonshot | Toolathlon; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.6 card: V* (w/ python) moonshot | V* (w/ python); vision; max tokens 98,304; avg@3; Python tool: 65,536 per step, 50 steps; temperature 1.0, top-p 1.0, context 262,144; Kimi K2.6 model card (HF created 2026-04-14); read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| Kimi K2.7 Code card: Kimi Claw 24/7 Bench moonshot | Kimi Claw 24/7 Bench; in-house; 17 scenarios/610 evaluation points; OpenClaw harness; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo |
| Kimi K2.7 Code card: Kimi Code Bench v2 moonshot | Kimi Code Bench v2; in-house; 10+ languages; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo |
| Kimi K2.7 Code card: MCP Atlas moonshot | MCP Atlas; 100 tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo |
| Kimi K2.7 Code card: MCP Mark Verified moonshot | MCP Mark Verified; human-verified MCPMark; 100-step tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo |
| Kimi K2.7 Code card: MLS Bench Lite moonshot | MLS Bench Lite; 30-task subset; 5 h per task; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo |
| Kimi K2.7 Code card: Program Bench moonshot | Program Bench; 200 tasks; 248k fuzz tests; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Kimi K2.7 Code·solo |
| Kimi K3 card: AA-Briefcase (Elo) moonshot | AA-Briefcase Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: AA-LCR moonshot | AA-LCR; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: APEX-Agents moonshot | APEX-Agents; Mercor leaderboard as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: AutomationBench moonshot | AutomationBench; 600-task public subset; official GitHub setup; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: BabyVision w/ python moonshot | BabyVision with Python; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: CharXiv (RQ) (no tools) moonshot | CharXiv reasoning questions; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: CharXiv (RQ) (with python) moonshot | CharXiv reasoning questions; avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: GDPval-AA v2 (Elo) moonshot | GDPval-AA v2 Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: GPQA Diamond moonshot | GPQA Diamond; no tools; top-p 0.95; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: HLE-Full (no tools) moonshot | HLE full set; top-p 0.95; with-tools = general tools; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: HLE-Full (with tools) moonshot | HLE full set; top-p 0.95; with-tools = general tools; with tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: MathVision (no tools) moonshot | MathVision; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: MathVision (with python) moonshot | MathVision; avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: MCP-Atlas moonshot | MCP-Atlas; 500-task public subset; 100-turn limit; judge Gemini 3.1 Pro; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: MCPMark-Verified moonshot | MCPMark-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: MMMU-Pro (no tools) moonshot | MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: MMMU-Pro (with python) moonshot | MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: MMVU moonshot | MMVU; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: OmniDocBench moonshot | OmniDocBench; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: OSWorld 2.0 moonshot | OSWorld 2.0; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: OSWorld-Verified moonshot | OSWorld-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: PerceptionBench moonshot | PerceptionBench (Moonshot in-house benchmark); avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: ResearchRubrics moonshot | ResearchRubrics; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: SaaS-Bench moonshot | SaaS-Bench; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: SciCode moonshot | SciCode; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: Toolathlon-Verified moonshot | Toolathlon-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: Video-MME (w. sub) moonshot | Video-MME with subtitles; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: WorldVQA ForceAnswer moonshot | WorldVQA ForceAnswer variant; vision avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: ZeroBench (pass@5) (no tools) moonshot | ZeroBench main set; pass@5 (run 5 times, official setting); no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: ZeroBench (pass@5) (with python) moonshot | ZeroBench main set; pass@5 (run 5 times, official setting); with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| Kimi K3 card: τ³-Banking moonshot | tau3-Banking; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| latitude-sweep github-latitude | — | xmodel | Kimi K3·max |
| lb2-browser-analysis-driven-action livebench | — | xmodel | Kimi K3·max |
| lb2-browser-constraint-adherence livebench | — | xmodel | Kimi K3·max |
| lb2-browser-data-aggregation livebench | — | xmodel | Kimi K3·max |
| lb2-browser-sequential-planning livebench | — | xmodel | Kimi K3·max |
| lb2-browser-structural-integrity livebench | — | xmodel | Kimi K3·max |
| lb2-coding-billing-limits livebench | — | xmodel | Kimi K3·max |
| lb2-coding-cloud-infrastructure livebench | — | xmodel | Kimi K3·max |
| lb2-coding-hard-fixes livebench | — | xmodel | Kimi K3·max |
| lb2-coding-llm-api-integration livebench | — | xmodel | Kimi K3·max |
| lb2-coding-memory-tools livebench | — | xmodel | Kimi K3·max |
| lb2-coding-product-ux livebench | — | xmodel | Kimi K3·max |
| lb2-dataanalysis-consecutive-events livebench | — | xmodel | Kimi K3·max |
| lb2-dataanalysis-table-join livebench | — | xmodel | Kimi K3·max |
| lb2-dataanalysis-table-reformat livebench | — | xmodel | Kimi K3·max |
| lb2-design-documents-docx livebench | — | xmodel | Kimi K3·max |
| lb2-design-pdf livebench | — | xmodel | Kimi K3·max |
| lb2-design-presentations-ppt livebench | — | xmodel | Kimi K3·max |
| lb2-longhorizon-delegated-operations livebench | — | xmodel | Kimi K3·max |
| lb2-longhorizon-finance-compliance-apps livebench | — | xmodel | Kimi K3·max |
| lb2-longhorizon-long-run-operations livebench | — | xmodel | Kimi K3·max |
| lb2-longhorizon-operations-commerce-apps livebench | — | xmodel | Kimi K3·max |
| lb2-longhorizon-performance-optimisation livebench | — | xmodel | Kimi K3·max |
| lb2-longhorizon-systems-engineering livebench | — | xmodel | Kimi K3·max |
| lb2-reasoning-logic-with-navigation livebench | — | xmodel | Kimi K3·max |
| lb2-reasoning-theory-of-mind livebench | — | xmodel | Kimi K3·max |
| lb2-research-legal livebench | — | xmodel | Kimi K3·max |
| lb2-research-regulatory-compliance livebench | — | xmodel | Kimi K3·max |
| lb2-research-science-evidence livebench | — | xmodel | Kimi K3·max |
| lb2-research-strategy livebench | — | xmodel | Kimi K3·max |
| lb2-research-tax-finance livebench | — | xmodel | Kimi K3·max |
| Lechmazur Elimination Game (TrueSkill mu) lechmazur-elimination-game | multi-player elimination tournament (public/private chat, votes, jury), TrueSkill mu aggregated over random passes; games per model printed; README leaderboard of 61 rows; changelog 'Jan 6, 2026: GPT-5.2, Opus 4.5, Gemini 3 Pro, Gemini 3 Flash, Kimi K2 Thinking, Qwen 3 Max Thinking, Grok 4.1 Fast ... added' (stale board); no cost; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| LiveBench 2.0 · Browser Use suite livebench | LiveBench 2.0 Browser Use suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| LiveBench 2.0 · Coding suite livebench | LiveBench 2.0 Coding suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| LiveBench 2.0 · Data Analysis suite livebench | LiveBench 2.0 Data Analysis suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| LiveBench 2.0 · Design suite livebench | LiveBench 2.0 Design suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| LiveBench 2.0 · Long-Horizon suite livebench | LiveBench 2.0 Long-Horizon suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| LiveBench 2.0 · Reasoning suite livebench | LiveBench 2.0 Reasoning suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| LiveBench 2.0 · Research suite livebench | LiveBench 2.0 Research suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| livebench-task-amps_hard livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-code_completion livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-code_generation livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-connections livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-consecutive_events livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-integrals_with_game livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-javascript livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-logic_with_navigation livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-math_comp livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-olympiad livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-paraphrase livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-plot_unscrambling livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-python livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-simplify livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-spatial livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-story_generation livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-summarize livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-tablejoin livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-tablereformat livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-theory_of_mind livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-typescript livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-typos livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| livebench-task-zebra_puzzle livebench | — | xmodel | Kimi K2 Thinking·solo · Kimi K2.6·think |
| LiveCodeBench-v6 (GLM-4.7 card) zhipu | LiveCodeBench-v6; GLM-4.7 card 2026-01-29; settings: default settings; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| llm-stats.com arena: agents category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), agents category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo |
| llm-stats.com arena: general category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), general category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo |
| llm-stats.com arena: reasoning category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), reasoning category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo |
| llm2014-logic-2026-10-median llm2014-logic | — | xmodel | Kimi K3·max |
| LLMConfBench arxiv-2609.20666 | arXiv 2609.20666 · conformer energy ranking, 27 molecules × 3 prompts via OpenRouter, Opus 5 and Sonnet 5 at high; cost summed from the repo's per-call logs ✓ | xmodel | Kimi K3·high |
| LMArena Agent lmarena.ai | Agent Mode real sessions (Code/Chat/Work) · cost=median $/task (14d window), score=Net Improvement composite signal, effort read from model-variant label; live board, read 27 Sep 2026 ✓ | xmodel | Kimi K3·max |
| LMArena Text lmarena.ai | Text arena (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-opus-5.5-high-vertex); live board, read 30 Sep 2026 ✓ | xmodel | Kimi K3·max · Kimi K2 Thinking·solo |
| LMArena Text (xAI Grok 4.1 post) xai | LMArena Text Arena Elo as printed by xAI (chart label 'Overall Style Control Elo', text 'overall'; style control unclear), live board snapshot of 17 Nov 2025; no cost; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| LMArena WebDev lmarena.ai | Code arena, WebDev board (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-sonnet-5-5-high); live board, read 30 Sep 2026 ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo · Kimi K2 Thinking·solo |
| LMArena WebDev — HTML sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-html, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo · Kimi K2 Thinking·solo |
| LMArena WebDev — React sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-react, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo · Kimi K2 Thinking·solo |
| MathArena · aime 2026 matharena.ai | MathArena, aime--aime_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| MathArena · apex 2025 matharena.ai | MathArena, apex--apex_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| MathArena · april matharena.ai | MathArena, arxiv--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · april matharena.ai | MathArena, arxiv_false--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · august matharena.ai | MathArena, arxiv--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · august matharena.ai | MathArena, arxiv_false--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · february matharena.ai | MathArena, arxiv--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K2.6·think |
| MathArena · february matharena.ai | MathArena, arxiv_false--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K2.6·think |
| MathArena · hmmt feb 2026 matharena.ai | MathArena, hmmt--hmmt_feb_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| MathArena · january matharena.ai | MathArena, arxiv--january: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K2.6·think |
| MathArena · june matharena.ai | MathArena, arxiv--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · june matharena.ai | MathArena, arxiv_false--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · kangaroo 2025 1-2 matharena.ai | MathArena, kangaroo--kangaroo_2025_1-2: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · kangaroo 2025 11-12 matharena.ai | MathArena, kangaroo--kangaroo_2025_11-12: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · kangaroo 2025 3-4 matharena.ai | MathArena, kangaroo--kangaroo_2025_3-4: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · kangaroo 2025 5-6 matharena.ai | MathArena, kangaroo--kangaroo_2025_5-6: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · kangaroo 2025 7-8 matharena.ai | MathArena, kangaroo--kangaroo_2025_7-8: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · march matharena.ai | MathArena, arxiv--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K2.6·think |
| MathArena · march matharena.ai | MathArena, arxiv_false--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K2.6·think |
| MathArena · may matharena.ai | MathArena, arxiv--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · may matharena.ai | MathArena, arxiv_false--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| MathArena · shortlist 2025 matharena.ai | MathArena, apex--shortlist_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| Mercor · Harvey LAB mercor.com | Mercor, Harvey LAB: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| Mercor · Long Context Reasoning (AA-LCR v1.1) mercor.com | Mercor, Long Context Reasoning (AA-LCR v1.1): the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| Mercor · SWE Atlas - Codebase QnA mercor.com | Mercor, SWE Atlas - Codebase QnA: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| Mercor APEX-Accounting mercor.com | Mercor's own board, APEX-Accounting, harness loop_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| Mercor APEX-Agents mercor.com | Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| Mercor APEX-v1 (The AI Productivity Index) mercor.com | Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K3·max |
| mercor-apexacct-mean mercor.com | — | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| mercor-apexv1-mean mercor.com | — | xmodel | Kimi K2.7 Code·solo · Kimi K3·max |
| MMLU-Pro (GLM-4.7 card) zhipu | MMLU-Pro; GLM-4.7 card 2026-01-29; settings: default settings temp 1.0 top-p 0.95 131072; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| Next.js Agent Evals (Vercel), base run nextjs-evals | 31 Next.js agent evals, 4 runs each (earlyExit), Claude Code via AI Gateway, no AGENTS.md; https://nextjs.org/evals, github.com/vercel/next-evals-oss@38cb8ad, agent-results.json metadata.experiments[].baseSuccessRate/avgCostUsd; read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| Next.js Agent Evals (Vercel), with AGENTS.md nextjs-evals | Same 31 evals/4 runs/Claude Code setup as nextjsevals-base, but with an AGENTS.md/CLAUDE.md file telling the agent the Next.js version has breaking changes; score = docsSuccessRate (board); cost = mean of per-eval costUsd in agent-results.json['results'][model] (computed-by-us, since the board's avgCostUsd field is the base-run figure, not the AGENTS.md-run figure — the per-eval array's own mean, 0.2456 for Opus 5.5, differs from the base 0.234, confirming the two are different runs); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| NonoBench nonobench.com | NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓ | xmodel | Kimi K3·max |
| Nuxt agent evals (nuxt/nuxt-evals), Claude Code via AI Gateway nuxt/nuxt-evals | agent-results.json exportedAt 2026-09-22; 31 Nuxt agent evals, runs: 4, earlyExit: true; score is avgPassRate (fraction of the 4 runs passing, averaged over 31 evals), *100 for a percentage; cost is the gateway's avg cost USD per eval run; models ran on different dates (opus-5.5 2026-09-22, the rest 2026-08-27 to 09-05); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| nuxt-evals-success nuxt/nuxt-evals | — | xmodel | Kimi K2.7 Code·solo |
| NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections) matsonj/eval-connections | results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓ | xmodel | Kimi K3·low |
| OpenRouter GPQA-Diamond rolling-mean board (snapshot 2026-10-03) openrouter | reasoning-math/hard, OR-internal rolling-mean harness, auto-routing provider aggregate; ref openrouter.ai/benchmarks/gpqa-diamond; SNAPSHOT REPLACES prior orgpqa snapshot (2026-09-27); read 2026-10-05 ✓ | xmodel | Kimi K2 Thinking·solo · Kimi K2.7 Code·solo |
| OpenRouter tau2-bench-airline board (snapshot 2026-10-03) openrouter | agentic-tool/mid, tau2-bench airline tasks, OR-internal harness, auto-routing provider aggregate; ref openrouter.ai/benchmarks/tau2-bench-airline; SNAPSHOT REPLACES prior ortau2 snapshot (2026-09-27); read 2026-10-05 ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K2 Thinking·solo |
| openvl-base-avg22 arxiv-2610.26860 | — | xmodel | Kimi K3·max |
| openvl-ours-avg22 arxiv-2610.26860 | — | xmodel | Kimi K3·max |
| OSWorld-Pro arxiv-2609.24890 | arXiv 2609.24890 · OSWorld harness, 250 turns, 4 Claude models all at max, tokens × OpenRouter list incl. cache ✓ | xmodel | Kimi K3·max |
| prinzbench (private legal-research + search benchmark by a single author), leaderboard of 18 Jul 2026 github-prinz-ai-prinzbench | prinzbench (github.com/prinz-ai/prinzbench README 'Last updated on July 18, 2026'): 33 author-written questions (25 legal research + 8 needle search) asked 3 times, pass@1, graded by the author without blinding; score = full score x/99; access through chat apps or terminals at different dates (Jan-Jul 2026); '*' = early access before public release; the author cautions that Pro models use parallel test-time compute; read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| reactbench reactbench | — | xmodel | Kimi K2.7 Code·solo |
| reactbench-fix reactbench | — | xmodel | Kimi K2.7 Code·solo |
| reactbench-write reactbench | — | xmodel | Kimi K2.7 Code·solo |
| ReLE 专业能力 (professional ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/专业能力.md (专业能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 医疗与心理健康 (medical/mental health) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/医疗与心理健康.md (医疗与心理健康); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 总分 (total) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/total.md (alldata); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 推理与数学计算 (reasoning/math) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/推理与数学计算.md (推理与数学计算); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 教育 (education) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/教育.md (教育); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 法律与行政公务 (law/admin) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/法律与行政公务.md (法律与行政公务); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 语言与指令遵从 (language/IF) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/语言与指令遵从.md (语言与指令遵从); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 通用能力 (general ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/通用能力.md (通用能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReLE 金融 (finance) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/金融.md (金融); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| ReplaySCM Block-order (100 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Block-order (100 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | Kimi K2 Thinking·solo |
| ReplaySCM Hidden-order Hid-Full (250 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Hidden-order Hid-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | Kimi K2 Thinking·solo |
| ReplaySCM Ord-Full (250 problems): HeldoutWorldExact replayscm-2605.08197 | ReplaySCM Table 1 Ord-Full (250 problems), executable Boolean SCM induction, direct generation no tools, one stored response per model/problem, temp 0.1 (0.1 where set); metric HeldoutWorldExact (other columns = same runs, rule 8); arXiv 2605.08197 (5 May 2026); read 2026-10-20 ✓ | xmodel | Kimi K2 Thinking·solo |
| Roboflow Playground Vision Evals, OCR (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: OCR; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-ocr) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Kimi K2.6·nothink |
| Roboflow Playground Vision Evals, overall (mean of tasks) with cost per sample roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; Overall = page's mean over the tasks scored (4/5 where OCR is missing); cost = 'Avg cost / sample' in USD from price x tokens at the pricing date 2026-10-19 ('~' = estimated from self-host/OpenRouter pricing); read 2026-10-20 ✓ | xmodel | Kimi K2.6·nothink |
| scorch-bench: artillery duel game vs 7 bots, Claude Code subagents (reasoning track), seed 17 github-scorch-bench | smerwin/scorch-bench bench/RESULTS-claude.md (2026-10-09); 8 matches per model on one seed; one fresh Claude Code subagent per round via relay provider; score = win rate %; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| SenseBench (WSD) · prompt p001 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Kimi K2.7 Code·solo |
| SimpleBench multiple-choice (AVG@5) with cost per run simplebench | SimpleBench >200 private multiple-choice questions (spatio-temporal, social intelligence, trick questions); score = Score (AVG@5) as printed; cost = modelMeta.cpr 'pre-computed cost to run the whole benchmark once (measured tokens x OpenRouter price)' USD; tout = measured output tokens (unit unstated); live board, rows added between 2025-10 and 2026-10-01 (dateAdded printed); read 2026-10-19 ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo · Kimi K2 Thinking·solo |
| SuperCLUE 2025年度测评 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| SuperCLUE 2025年度测评 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| SuperCLUE 2025年度测评 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| SuperCLUE 2025年度测评 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| SuperCLUE 2025年度测评 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| SuperCLUE 2025年度测评 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2025年度测评, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| SuperCLUE 2026年5月 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| SuperCLUE 2026年5月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| SuperCLUE 2026年5月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| SuperCLUE 2026年5月 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| SuperCLUE 2026年5月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| SuperCLUE 2026年5月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Kimi K2.6·think |
| SuperCLUE 2026年7月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| SuperCLUE 2026年7月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| SuperCLUE 2026年7月 智能体任务规划 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体任务规划, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| SuperCLUE 2026年7月 智能体编程 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 智能体编程, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| SuperCLUE 2026年7月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| SuperCLUE 2026年7月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年7月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; cost = SuperCLUE 2026年7月_成本 sheet (same 18 models/effort strings/builds as this board), CNY, tokens x list unit price; read 2026-10-19 ✓ | xmodel | Kimi K3·max |
| SuperCLUE-Terminal (Chinese agentic terminal coding) superclue-terminal | SuperCLUE-Terminal (CLUE team, Aug 2026; primary board/WeChat article NOT reached): Chinese-language real programming tasks of 50-110 interaction rounds each, all models through a Claude Code-based generic agent framework; numbers read from two secondary pages dated 14 Aug 2026 (we0.ai, news.aibase.com) that agree; cost = reported mean per-task cost in CNY (benchmark execution cost, not provider pricing); read 2026-10-20 ✓ | xmodel | Kimi K3·max |
| Surface Evolver bench yhenon-surface-evolver | yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓ | xmodel | Kimi K2.7 Code·solo |
| surfevolver-pass yhenon-surface-evolver | — | xmodel | Kimi K2.7 Code·solo |
| Surge AI · Antidote: Everyday Edition surgehq.ai | Surge AI Antidote: Everyday Edition leaderboard (surgehq.ai/benchmarks/antidote); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| Surge AI · Chartography surgehq.ai | Surge AI Chartography leaderboard (surgehq.ai/benchmarks/chartography); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| Surge AI · ComplexConstraints surgehq.ai | Surge AI ComplexConstraints leaderboard (surgehq.ai/benchmarks/complex-constraints); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| Surge AI · DAYJOB: Finance surgehq.ai | Surge AI DAYJOB: Finance leaderboard (surgehq.ai/benchmarks/dayjob-finance); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| Surge AI · DAYJOB: Healthcare surgehq.ai | Surge AI DAYJOB: Healthcare leaderboard (surgehq.ai/benchmarks/dayjob-healthcare); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| Surge AI · EnterpriseBench: CoreCraft surgehq.ai | Surge AI EnterpriseBench: CoreCraft leaderboard (surgehq.ai/benchmarks/enterprisebench-corecraft); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| Surge AI · GDP.pdf surgehq.ai | Surge AI GDP.pdf leaderboard (surgehq.ai/benchmarks/gdp-pdf); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| Surge AI · GDP.xlsx surgehq.ai | Surge AI GDP.xlsx leaderboard (surgehq.ai/benchmarks/gdp-xlsx); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| Surge AI · HANDBOOK.md Agents surgehq.ai | Surge AI HANDBOOK.md Agents leaderboard (surgehq.ai/benchmarks/handbook); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K2.6·think · Kimi K3·max |
| Surge AI · Hemingway-bench surgehq.ai | Surge AI Hemingway-bench leaderboard (surgehq.ai/benchmarks/hemingway-bench); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| Surge AI · Riemann-bench surgehq.ai | Surge AI Riemann-bench leaderboard (surgehq.ai/benchmarks/riemann-bench); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Kimi K3·max · Kimi K2.6·think |
| surge-twi-oct9 surge-twi | — | xmodel | Kimi K3·max |
| SVGBench README leaderboard · score % svgbench-github | SVGBench (github.com/johnbean393/SVGBench) README leaderboard at commit 55c1cde8 (2026-02-12): single score column, 100+ SVG-generation questions with requirement-based scoring; effort = the label in parentheses where present; read 2026-10-20 ✓ | xmodel | Kimi K2 Thinking·solo |
| SWE-bench Verified (GLM-4.7 card) zhipu | SWE-bench Verified; GLM-4.7 card 2026-01-29; settings: temperature 0.7 top-p 1.0 max new tokens 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| tau2-Bench (GLM-4.7 card) zhipu | tau2-Bench; GLM-4.7 card 2026-01-29; settings: temperature 0 max new tokens 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| TaxCalcBench TY25, no web search (column-tax/tax-calc-bench) column-tax-tax-calc-bench | 50 TY25 federal+state tax-return cases, 1 run each; strict-correct-return % is the score (lenient% also in the evidence but not used here per instructions); cost = mean litellm-estimate USD/return (n priced noted when <50); source: taxcalc_ty25_claude_by_effort.txt, re-verified against raw evaluation_result_*.md for opus-5.5/ultrathink (40.00%/50.00%/$1.135, exact match); read 2026-10-01 ✓ | xmodel | Kimi K3·max |
| TaxCalcBench TY25, with web search (column-tax/tax-calc-bench) column-tax-tax-calc-bench | Same as taxcalc-ty25-notool but with the web-search tool enabled; read 2026-10-01 ✓ | xmodel | Kimi K3·max |
| Terminal Bench 2.0 (GLM-4.7 card) zhipu | Terminal Bench 2.0; GLM-4.7 card 2026-01-29; settings: temperature 0.7 top-p 1.0 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| Terminal Bench Hard (GLM-4.7 card) zhipu | Terminal Bench Hard; GLM-4.7 card 2026-01-29; settings: temperature 0.7 top-p 1.0 16384; footnotes on judges/harness not in the card; read 2026-10-19 ✓ | xmodel | Kimi K2 Thinking·solo |
| Tinybird SQL benchmark llm-benchmark.tinybird.live | Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓ | xmodel | Kimi K2.7 Code·solo |
| Toolathlon-Verified leaderboard (current, 108 tasks) toolathlon | Toolathlon-Verified (released 30 Jun 2026), 108 tasks, 32 MCP servers / 604 tools + 7 local toolkits, Pass@1 % (+-CI) over 3 runs; rows carry the 'Evaluated by us' badge unless noted; Agent 'Default'; page fetched 2026-10-20 (newest row 2026-08-30); cells identical to held cells (vendor cards/blogs copying the board) dropped; read 2026-10-20 ✓ | xmodel | Kimi K3·max · Kimi K2.7 Code·solo |
| tracker-quality-score github-startakovsky-llm-model-tracker | — | xmodel | Kimi K2.7 Code·solo |
| UGI Leaderboard — UGI composite (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, composite UGI score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Kimi K2 Thinking·solo |
| UGI Leaderboard — World Model factual recall (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, recipe/geoguessr/weight/music percent-error battery; test dates vary by model; read 2026-10-05 ✓ | xmodel | Kimi K2 Thinking·solo |
| UGI Leaderboard — Writing quality (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, LLM-judged Writing score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Kimi K2 Thinking·solo |
| Vals AIME vals.ai | Vals AI · AIME, compute_effort null (effort unverified), board 2026-04-16 ✓ | xmodel | Kimi K2 Thinking·solo |
| Vals CaseLaw v2 vals.ai | Vals AI · CaseLaw v2, compute_effort null (effort unverified), board 2026-05-04 ✓ | xmodel | Kimi K2 Thinking·solo |
| Vals Code Migration vals.ai | Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K3·max |
| Vals CorpFin v2 vals.ai | Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Kimi K3·max |
| Vals CyberBench vals.ai | Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals Finance Agent v2 vals.ai | Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals GPQA vals.ai | Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Kimi K3·max |
| Vals HLAB vals.ai | Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals Index (29 Sep) vals.ai | vals.ai Vals Index as recomputed on 29 Sep 2026 (older models dropped), max effort, accuracy and cost per test ✓ | xmodel | Kimi K3·max |
| Vals IOI vals.ai | Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals Legal Research vals.ai | Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals LegalBench vals.ai | Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals LiveCodeBench vals.ai | Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K3·max |
| Vals MedCode vals.ai | Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓ | xmodel | Kimi K3·max |
| Vals MedQA vals.ai | Vals AI · MedQA, Sonnet 4.6 vs Haiku 4.5 thinking, effort null, board 2026-04-16 ✓ | xmodel | Kimi K2 Thinking·solo |
| Vals MedScribe vals.ai | Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Kimi K3·max |
| Vals MMLU-Pro vals.ai | Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Kimi K3·max |
| Vals MMMU vals.ai | Vals AI · MMMU, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Kimi K3·max |
| Vals MortgageTax vals.ai | Vals AI · MortgageTax, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Kimi K3·max |
| Vals ProgramBench vals.ai | Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K3·max |
| Vals ProofBench vals.ai | Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals Public Benefits v1.1 vals.ai | Vals AI · Public Benefits v1.1, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K3·max |
| Vals SAGE vals.ai | Vals AI · grading student math work against a rubric, all at max ✓ | xmodel | Kimi K3·max |
| Vals SWE-bench vals.ai | Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Kimi K2.7 Code·solo · Kimi K3·max |
| Vals TaxEval v2 vals.ai | Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Kimi K3·max |
| Vals TB 2.1 vals.ai | Vals AI · Terminal-Bench 2.1, Claude models at their stated compute_effort (Sonnet 5.5, Opus 5.5 high) — snapshot 27 Sep, re-priced ✓ | xmodel | Kimi K2.7 Code·solo |
| Vals Time Horizon KSP vals.ai | Vals AI · Time Horizon Index: KSP, 3 Claude at max, board 2026-09-14 ✓ | xmodel | Kimi K3·max |
| vals.ai SWE-bench Multimodal (JS front-end issues with screenshots) vals.ai | Vals-run agent; resolve rate overall, cost per test (USD); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓ | xmodel | Kimi K3·max |
| valscodemig11 vals.ai | — | xmodel | Kimi K3·max |
| valsemb11 vals.ai | — | xmodel | Kimi K3·max |
| valsfab21 vals.ai | — | xmodel | Kimi K3·max |
| valsfinindex vals.ai | — | xmodel | Kimi K3·max |
| valshlab-2026-10-21 vals.ai | — | xmodel | Kimi K3·max |
| valsindex4 vals.ai | — | xmodel | Kimi K3·max |
| valsinfprovider vals.ai | — | xmodel | Kimi K3·max |
| valslegalindex vals.ai | — | xmodel | Kimi K3·max |
| valssage11 vals.ai | — | xmodel | Kimi K3·max |
| valstaxwp vals.ai | — | xmodel | Kimi K3·max |
| Vending-Bench 2 (Andon Labs) andonlabs-vb2 | andonlabs.com/evals/vending-bench-2: mean final money balance after a simulated year (start $500), mean of runs; 3 rows read from the page, the rest relayed by Epoch's mirror; no effort stated by the page; no cost; read 2026-10-15 ✓ | xmodel | Kimi K2.7 Code·solo |
| vlm-exam counting github-vlmexam | roboflow/vlm-exam · 74 samples, LLM-judge accuracy ✓ | xmodel | Kimi K2.6·nothink |
| vlm-exam detection github-vlmexam | roboflow/vlm-exam · 250 images, mAP@50 ✓ | xmodel | Kimi K2.6·nothink |
| vlm-exam extraction github-vlmexam | roboflow/vlm-exam · 97 samples, LLM-judge accuracy; same models as OCR ✓ | xmodel | Kimi K2.6·nothink |
| vlm-exam identification github-vlmexam | roboflow/vlm-exam · 32 samples, near ceiling ✓ | xmodel | Kimi K2.6·nothink |
| vlm-exam OCR github-vlmexam | roboflow/vlm-exam · 37 samples, similarity; Fable 5.1 & Opus 5.5 low/high ×3 runs, others low ×1 ✓ | xmodel | Kimi K2.6·nothink |
| vlmexam-text github-vlmexam | — | xmodel | Kimi K2.6·nothink · Kimi K3·high |
| vulcanbench3 github-vulcanbench | — | xmodel | Kimi K3·max |
| τ^τ-Bench arxiv-2609.04611 | arXiv 2609.04611 · 53 agent-construction tasks in Claude Code, Opus 5 vs Sonnet 5 both at max, build spend at list ✓ | xmodel | Kimi K3·max |
How the numbers are built — the method
Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.
In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.
The procedure
- Every number traced to its source. Each source is recorded as facts: the value as the source writes it, the exact text or table cell it is read from, the URL and the date of the capture; agents read every value line by line, and a measurement whose facts lack their proof stays out. Registries of models (the names each was served under, its effort settings and defaults, dated), routes (makers' APIs, resellers, routers, harnesses), benchmarks and their versions, publishers and dated prices let one build decide, by a fixed decision tree, which model, configuration and price each measurement belongs to.
- Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale, one version of a board; a group that mixes them is split. Every run is dated, and its model and effort are read as the maker and every route in between served them on that date — never assumed: a route the source does not name is never presumed to be the maker's API, and every route that could have served the run must give the same configuration. At the least doubt on the model, the effort or what was run, the measurement stays in the data file and out of the fit.
- Costs at today's prices. Every cost is put on the model's current price from the prices in force on its run date: the source's own figure at that price, the price change applied to the run's own token counts, or, when the token mix is unknown, a price band whose width is added to the cost's precision. A cost that cannot be brought to the current price stays out.
- Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The scale is read from the metric's label, never from the benchmark's identity; whether lower is better is read from the benchmark's registry, and such a score is turned in its own unit first (an attack success rate of 30 % reads 70 %); a score whose direction is not established stays out. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
- One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result. The sizes of these effects are estimated for each model maker on its own, not tied to the other makers'; a maker's own charts about its own models carry an estimated home effect, also each maker's own, left out of every value. The fit takes the couples of eight makers measured in the same groups (Anthropic, OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi), on one scale; this page shows the Kimi couples. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
- Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source; the one weight set by rule is that of a single run: when a source gives k quality measurements of one run (several metrics, sub-scores or slices of the same executions), each weighs 1/k, so the run counts as one measurement.
- Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, a board copying another's numbers, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. Versions of one board that re-grade the same runs (recognised by identical costs) count those runs once. Before the fit, two groups that still share identical costs on three couples or more stop it until the copy is resolved or the coincidence proven from the sources. An index keeps, for a couple whose index components are already in the data, only what its missing components add, or is left out.
- Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
- From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.
Key values
- Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
- Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
- Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit one figure on this page can move by one unit in that digit (the cost of Kimi K3 at high), far inside their intervals.
- Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.
Techniques and software
- Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes, six chains whose draws continue in the same run until the criteria are met (CmdStan only to resume a stopped fit from its stored draws, or to restart a chain left in a far region from a good chain's last draw); built and checked stage by stage following the Bayesian workflow.
- Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
- Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
- Scales: the empirical logit of bounded scores, with the binomial shape of its noise.
Checks (on the fit of 29 September 2026, before the other makers' couples joined the data). Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.