What we do and what we do not do
LLMBenchmark.ai aggregates. We collect results that other people publish, match each row to a model in our catalog, and compute an index across them. We run no benchmark, so every number on the site traces to a public leaderboard, and each page names the source and the day we read it.
List prices come from OpenRouter. They are API prices per million tokens, not subscription prices.
What goes into the index
Each result is rescaled to 0 to 100 and the results are averaged with the weights in the table. The weights are our judgment, not a measured quantity. They are printed here so you can recompute any score.
| Benchmark | Weight | Rescaled from |
|---|---|---|
| Aider polyglot | 35% | 0% to 100% |
| SWE-bench Verified | 30% | 0% to 100% |
| LiveBench coding | 20% | 0% to 100% |
| LMArena WebDev | 15% | 1200 to 1900 |
The ranges are fixed. They are not taken from today's best and worst model, so a new leader does not move anyone else's score. A result outside its range is clamped to the nearest end.
When a source lists one model several times, for example at different reasoning efforts or agent setups, we keep its best published run. The index shows what a model can do when well configured, not what a default setup gives.
A model measured on only some benchmarks is scored on those. The weights of the benchmarks it has are scaled to add up to 100%, so a missing benchmark is not counted as a zero.
Coverage: how many benchmarks stand behind a score
Coverage counts the distinct weighted benchmarks with a result for a model. Three or more is verified, two is partial, one is thin, none is no coverage. A thin score can be right, but it rests on one measurement and the dots say so.
Of 356 models, 2 have three independent coding benchmarks, 64 have two, 70 have one and 220 have none.
The ranking sorts by coverage first and by index second. A score backed by three benchmarks outranks a higher score backed by two, because the second one has not earned the position yet.
Collected, shown, and left out of the index
We also collect the Vals Index from vals.ai, 6 models matched. It appears on model pages as supporting evidence, and the index deliberately excludes it. vals.ai measures general capability across professional tasks. This index is specifically about coding, and mixing the two would make it answer a different question.
Three more single boards from Vals AI stand outside coding: GPQA Diamond for reasoning, AIME for math and MMLU Pro for knowledge. Each one is its own ranking, shown as published with the date the board gives and the limits Vals AI states about it, and none of them feeds the index. With one board per axis there is nothing to average, so we do not build a score for those axes.
Missing values are never interpolated
When a model has no result on a benchmark, its page says "Not measured". We do not estimate the result from the maker's other models, from the price, or from a benchmark that tends to move with it. The same holds for price and for cost per solved task, where the label says why the number is absent.
Cost per solved task, and how it differs from Cost per Task
Artificial Analysis publishes a Cost per Task, which its site describes as the weighted average cost in USD per Intelligence Index task. That is spend per task the model is given. Our cost per solved task is spend per task the model solves, the cost of one attempt divided by the share of attempts that succeed, so a model that is cheap per attempt and fails often does not look cheap on ours.
Both numbers come from one run of SWE-bench Verified. Vals AI publishes the cost per test and the score for the current models it runs on one shared harness, and we read those first; swebench.com fills in models Vals AI has not run. We never divide one board's cost by the other board's pass rate. Where a board publishes what its own tests cost, its page shows that cost beside the result, exactly as published.
What the index cannot tell you
- Public benchmarks can leak into training data, so a high score can overstate real ability. We cannot measure by how much.
- The boards update on their own schedules. Each source below carries the day we read it.
- It ranks general coding ability on these tasks. Your codebase, language and prompts will rank models differently.