SWE-bench Verified explained: the coding test that grades real GitHub fixes.
A model’s SWE-bench Verified score is the share of 500 real, human-checked GitHub issues it can fix end to end. Here is how the test works, how to read the numbers on our leaderboard, and where they mislead.
By AI Pro League · Published
SWE-bench Verified is the coding benchmark this site weights most heavily after general capability: it carries 20% of the League Score. It is also the most misread number in AI — a single percentage that hides a very specific test. This explainer uses our own SWE-bench Verified leaderboard as the worked example throughout.
What SWE-bench Verified measures
The original SWE-bench hands a model a real open-source Python repository plus the text of a real GitHub issue, and asks for a patch. It contains 2,294 issues drawn from a dozen Python repositories. Verified is the human-verified 500-task subset of that pool: every task was checked by hand to confirm the issue is solvable and the tests are fair. A task counts as resolved when the model’s patch makes the tests that were failing on the issue pass, without breaking tests that already passed. The score is simply the percentage of the 500 tasks resolved. The benchmark’s home is swebench.com; the scores we record come from published leaderboards such as BenchLM’s SWE-bench Verified leaderboard, always with the source and the date we verified them.
How a SWE-bench Verified score is calculated
Two different calculations matter. The benchmark score itself is a plain resolution rate: resolve 250 of the 500 tasks and the score is 50. Our League Score then normalises that raw score across the field we track — today the recorded range runs from 51.0 to 96.0 — and weights the result at 20%. The worked example:
| Model | Raw score | Normalised (0–100) |
|---|---|---|
| Claude Opus 5 | 96.0 | (96.0 − 51.0) / (96.0 − 51.0) × 100 = 100.0 |
| Claude Sonnet 5 | 85.2 | (85.2 − 51.0) / (96.0 − 51.0) × 100 = 76.0 |
| Kimi K2.6 | 80.2 | (80.2 − 51.0) / (96.0 − 51.0) × 100 = 64.9 |
Normalisation is why a raw gap of a few points near the top of the table can be worth more League Score than a big gap at the bottom — and why the component is read against the current field, not against an absolute standard. Full method: how the League Score is built.
What a good SWE-bench Verified score looks like
Only 13 of the 59 models in our database carry a recorded score, and the spread is wide: the median recorded score is 73.6, the floor is 51.0 and the ceiling is 96.0. Until more of the field is measured, treat the ranking as a leaderboard of the tested, not of everyone.
| # | Model | Maker | Score | Measured |
|---|---|---|---|---|
| 1 | Claude Opus 5 | Anthropic | 96.0 | Oct 14, 2026 |
| 2 | Claude Sonnet 5 | Anthropic | 85.2 | Oct 14, 2026 |
| 3 | Kimi K2.6 | Moonshot AI | 80.2 | Oct 14, 2026 |
| 4 | Mistral Medium 3.5 | Mistral AI | 77.6 | Apr 29, 2026 |
| 5 | Qwen3.5-397B-A17B | Alibaba Qwen | 76.2 | Oct 14, 2026 |
The full table, with a source link on every row, is the SWE-bench Verified leaderboard. The shortlist it feeds is our best AI models for coding page.
The limits to keep in mind
- Coverage is partial. Most tracked models have no recorded score, so absence from the table means untested, not weak.
- It is Python-only. The underlying issues come from Python repositories; the score says nothing direct about other languages.
- Scores age. Our recorded results were measured on different dates, and the harness around a model can move its number — which is why every row on our leaderboard carries its source and date.
- It is one component. At 20% of the League Score it shapes the coding standings heavily, but general capability (LiveBench, 40%) still dominates the overall table.
For the wider map — what MMLU, HumanEval and the other famous tests measure — see our guide to AI benchmarks explained.