SWE-bench Verified Leaderboard 2026: Top Models Ranked (+ SWE-bench Pro)

Published October 30, 2025 • Updated October 15, 2026 • 14 min read

SWE-bench Verified leaderboard, the short answer: there are two boards that both call themselves "SWE-bench Verified", and they disagree. The official swebench.com board (500 tasks, submissions through February 26, 2026) tops out at 79.2%, held jointly by Sonar Foundation Agent + Claude 4.5 Opus and live-SWE-agent + Claude 4.5 Opus; the swebench.com team's own bash-only runs on February 17, 2026 put Claude 4.5 Opus (high) at 76.8%, Gemini 3 Flash and MiniMax M2.5 at 75.8% and Claude 4.6 Opus at 75.6%. The vals.ai board (updated September 1, 2026, a minimal bash-tool harness) ranks Claude Opus 5 at 97.0%, DeepSeek V4 Pro at 96.4%, GPT-5.6 Sol at 96.2% and Grok 4.6 at 95.6%, and states the benchmark has saturated. On the harder SWE-bench Pro, Scale's standardized public board (read October 15, 2026) tops out at 61.5% (Muse Spark 1.1), while vendor-reported Pro scores aggregated on October 14, 2026 go as high as 89.9% (Claude Opus 5.5). Which number you quote depends on the harness, and this page shows all four with their dates.

Why this benchmark matters: when Princeton researchers released SWE-bench in October 2023 (Jimenez et al., arXiv 2310.06770), the best model, Claude 2, resolved 1.96% of its 2,294 real GitHub issues. Unlike "write a function" tests, SWE-bench hands a model a whole repository and an issue text and asks for a patch that makes hidden tests pass. The rest of this page explains each variant (Verified, Lite, Full, Multilingual, Multimodal, Pro), what the leaderboards say today, what HumanEval and MBPP add, and how to test models on your own code.

Quick summary: the major AI coding benchmarks at a glance

Official swebench.com figures come from the site's published data file (github.com/swe-bench/swe-bench.github.io, data/leaderboards.json) read on October 15, 2026. Most top Verified entries there are "unchecked" (not re-run by the maintainers); the checked flag is noted in the table below where it matters. The HumanEval and MBPP rows are blank on purpose: neither has an official leaderboard and this page will not invent one.

Understanding these benchmarks is the groundwork for choosing an AI coding model.

---

What is SWE-bench? Real GitHub issues, not toy problems

SWE-bench was introduced in the paper "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (arXiv 2310.06770, submitted October 10, 2023) by Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan, a Princeton-led group. It collects 2,294 problems from real GitHub issues across 12 popular Python repositories. Each task gives a model the repository at the commit before the fix and the issue text, and asks for a patch. The patch passes if the tests that failed before now pass (fail-to-pass) and the tests that passed before still pass (pass-to-pass).

How a SWE-bench task works

1. Read the issue as the reporter wrote it, which is often vague or incomplete.
2. Navigate the repository, which can be thousands of files, to find the relevant code.
3. Write a patch that resolves the issue.
4. Survive the test suite: the hidden fail-to-pass tests must now pass, and nothing that passed before may break.

The paper's headline result was how badly the 2023 models did: "The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues." That number is the baseline everything below should be read against.

The family of SWE-bench variants

Sizes are from each variant's own page or paper: Lite, Multilingual, Multimodal, Pro.

---

SWE-bench vs SWE-bench Verified

SWE-bench Verified is a 500-task subset of the original test set, published by OpenAI in August 2024 (announcement) with the SWE-bench authors. The reason it exists is that the original set contained tasks that were unsolvable as written.

How the screening was done, from the announcement as quoted by Epoch AI and vals.ai:

• 93 software developers experienced in Python screened the samples.
• Each sample was labelled three times by separate annotators, and the highest-severity label of the three was kept.
• 38.3% of samples were flagged for underspecified problem statements.
• 61.1% were flagged for unit tests that could unfairly mark a valid solution wrong.
• 68.3% of the original samples were filtered out for one of those reasons or others.
• GPT-4o's score went from 16% on the original set to 33.2% on Verified with the best open-source scaffold at the time (Agentless), which shows how much of the original "difficulty" was task quality rather than model capability.

What Verified changes: scores on Verified are higher than on Full, not lower, because the unsolvable tasks are gone. A model that resolves 75% of Verified is resolving 75% of tasks that humans confirmed are solvable and fairly tested. The 2026 official boards show the gap: 79.2% top on Verified against 52.62% on Full.

Where Verified falls short in 2026: all 500 tasks are Python, they were public before most current models were trained, and both vals.ai and Epoch note the scores clustering near the ceiling. That is the motivation for SWE-bench Pro and Multilingual, covered below.

---

SWE-bench Verified leaderboard (two boards, two answers)

There is no single "official SWE-bench Verified score" for a model. Every result is for a model plus a scaffold (the agent loop, tools and retries around it), and different boards use different scaffolds. Here are the two boards most people mean, each with its date.

The official swebench.com board (submissions through February 26, 2026)

Source: swebench.com data file, 180 Verified entries, read October 15, 2026. "Checked" means the maintainers re-ran the submission; most top entries are self-reported. Models released after February 2026 (Claude Opus 5, GPT-5.6, DeepSeek V4, Grok 4.6) have no entry on this board.

The vals.ai board (updated September 1, 2026)

vals.ai runs every model through the same minimal harness: the model gets a single bash tool and 500 isolated Docker containers. The page states that "performance has saturated" and that new model releases are no longer evaluated.

Source: vals.ai SWE-bench Verified leaderboard, September 1, 2026. These are not comparable with the swebench.com rows above: different harness, different date, different models.

Why the two boards differ by almost 20 points

Three reasons, none of them a mistake:

1. Different models. The official board stops at February 2026 models; vals.ai includes the July-August 2026 generation.
2. Different harness. vals.ai's bash-only loop is one fixed scaffold; swebench.com entries are whatever each submitter built, and the maintainers' own bash-only runs there (75.6-76.8%) are closer to what the same models score elsewhere.
3. Different checking. The official board marks most top entries as unchecked; vals.ai runs everything itself.

Treat the swebench.com board as the canonical record of systems and vals.ai as a consistent measurement of models. Quote the board and the date with any number.

Earlier milestones on Verified (sourced points only)

The query "o3 SWE-bench Verified score" brings many readers here: the answer is 69.1% by OpenAI's own April 2025 figure, which led the board at launch and now sits roughly in the middle of the vals.ai table's history.

---

What a SWE-bench Verified score means in practice

A Verified score is the share of 500 human-screened Python issues a system resolved on one attempt, in a sandbox, with the tests hidden. Reading it:

• It is a system score. Model plus scaffold plus tool set plus turn budget. The same model moves several points between harnesses, which is why the two boards above disagree.
• Mid-90s on vals.ai means the Python issue-fixing task is close to solved for that harness, not that the model writes production code unaided. vals.ai itself says the benchmark has saturated.
• 70-79% on the official board is where the strongest open scaffolds sat in early 2026, and where a fresh self-submission of a current model lands before heavy scaffold tuning.
• The tasks are public. They were on GitHub before the models were trained. Scale AI built SWE-bench Pro partly to address that.
• Python only. For other languages, use SWE-bench Multilingual (below).

For a cross-vendor view of which closed models to try first, see the best AI coding models guide.

---

SWE-bench Pro: the harder benchmark that still separates models

SWE-bench Pro was published by Scale AI in September 2025 ("SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?", arXiv 2509.16941). Per Scale's leaderboard page it has 1,865 tasks across 41 professional repositories in Python, Go, JavaScript and TypeScript, split into a public set (731 tasks, GPL-licensed repos), a commercial set (276 tasks from startup partners' proprietary code) and a held-out set (858 tasks). A task is resolved only if the fail-to-pass tests pass and the pass-to-pass tests still pass, scored on one attempt.

The scaffolding caveat: standardized vs vendor-reported

Two families of SWE-bench Pro numbers circulate and they are not comparable:

1. Standardized (Scale SEAL): Scale runs every model through the same SWE-Agent harness (capped at 50 turns, or 250 uncapped; entries marked with an asterisk used the mini-swe-agent harness). This isolates the model from scaffold engineering.
2. Vendor-reported: the provider runs its own tuned agent and publishes the result. These run higher, sometimes much higher, for the same model.

Use the standardized board to compare models against each other. Use vendor numbers only to track one vendor's progress generation over generation, where the scaffold is held constant.

Scale SEAL SWE-bench Pro public leaderboard (read October 15, 2026)

Source: labs.scale.com, SWE-bench Pro public set. Asterisked rows ran with the mini-swe-agent harness. The page does not print a last-updated date; the figures are as displayed on October 15, 2026.

Vendor-reported SWE-bench Pro (aggregated October 14, 2026)

Source: llm-stats SWE-bench Pro aggregator, October 14, 2026, which collects each vendor's self-reported figure. We have not verified these against the vendors' own posts. Note that the same benchmark reads 61.5% at the top of Scale's standardized board and 89.9% at the top of this one.

---

Multilingual, Multimodal, Lite and Full: the other official boards

SWE-bench Multilingual (swebench.com, February 2026)

Per its page, Multilingual has 300 curated tasks from 42 repositories in nine languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust). The published baseline is Claude 3.7 Sonnet with SWE-agent at 43%, with per-language resolution from 28.57% (C/C++) to 58.14% (Rust). The current board, all entries checked by the maintainers:

If you do not work in Python, this is the board to read first. GLM 5's fourth place is the open-weight high-water mark on an official SWE-bench board.

SWE-bench Multimodal (swebench.com, 2025)

The Multimodal paper (October 2024) built 617 tasks from 17 JavaScript libraries for web UI, diagramming, data visualisation, syntax highlighting and maps; every task has at least one image in the issue or tests. The board is far lower than Verified: GUIRepair + o3 and Codefuse_Pycfuse_SVR share the top at 35.98% (July 1 and November 17, 2025), with OpenHands-Versa + Claude Sonnet 4 at 34.43%. Visual front-end work remains the least-solved part of the family.

SWE-bench Lite and Full

Lite (300 test tasks plus 23 dev, per its page) was filtered to drop tasks with images, external links, multi-file edits or vague descriptions, so it is cheaper to run. Top: ExpeRepair-v1.0 + Claude 4 Sonnet at 60.33% (June 25, 2025); the maintainers' checked SWE-agent + Claude 4 Sonnet run is 56.67%.

Full is the original 2,294-task test set. Top: Sonar Foundation Agent + Claude 4.5 Opus at 52.62% (December 19, 2025); the checked SWE-agent 1.0 + Claude 3.7 Sonnet run is 33.83%. Compare that with Claude 2's 1.96% in the 2023 paper.

---

HumanEval and MBPP: function-level benchmarks

HumanEval comes from OpenAI's Codex paper, "Evaluating Large Language Models Trained on Code" (arXiv 2107.03374, July 2021). It has 164 hand-written problems, each a Python function signature plus docstring; the model writes the body and hidden unit tests check it. The paper reported Codex solving 28.8% at pass@1 and 70.2% when allowed 100 samples per problem.

MBPP (Mostly Basic Programming Problems) comes from "Program Synthesis with Large Language Models" (arXiv 2108.07732, August 2021, Austin, Odena et al.) and has 974 entry-level Python tasks written to be solvable by beginners.

pass@k, explained

pass@k is the probability that at least one of k sampled solutions passes the tests. pass@1 is the number vendors quote. Pass@10 or pass@100 are useful for completion tools that show several suggestions, and show that models can often find a correct answer that they do not rank first.

Where the numbers come from today

Neither benchmark has a maintained official leaderboard, so scores are taken from vendors' model cards and differ in prompting and sampling. An example of a properly attributed row, from the Qwen2.5 release post: Qwen2.5-7B-Instruct 84.8 HumanEval / 79.2 MBPP, against the same team's runs of Llama-3.1-8B-Instruct (72.6 / 69.6) and Gemma2-9b-IT (68.9 / 74.9). Frontier models cluster in the 90s on HumanEval, which is why nobody uses it to rank them any more.

HumanEval vs SWE-bench: what each predicts

HumanEval / MBPP test writing a small function from a clear description with no surrounding code: algorithmic thinking, language syntax, edge cases.

SWE-bench tests reading an unfamiliar codebase, locating the cause of a vaguely described bug, and changing it without breaking anything else.

A model can score 90 on HumanEval and 40 on SWE-bench Pro; the second number is the one that predicts how it will behave in your repository. If you are choosing a model for maintenance work, weight SWE-bench; if for scripts and utilities, HumanEval-style results are still informative.

---

How to interpret benchmark scores

What the scores do tell you

• Relative capability under one harness. Within a single board, higher is better.
• Trend over time. Verified went from 33.2% (GPT-4o, 2024) to 97.0% (vals.ai, 2026); Full from 1.96% to 52.62%.
• Where a model class stands. A model near the top of Scale's standardized Pro board is a frontier coder regardless of its Verified number.

What they do not tell you

1. Your language. Verified is Python; Multilingual covers nine more; Multimodal is JavaScript with images. TypeScript-heavy teams should read Multilingual and Pro.
2. Your scaffold. The same model varies by several points between harnesses, and far more on Pro.
3. Code quality. A patch that passes the tests may still be one you would reject in review. No SWE-bench variant scores readability or maintainability.
4. Interaction. Benchmarks are one-shot. Real work is a conversation with clarifications.
5. Contamination. Verified tasks are public and old; Pro's held-out and commercial sets exist for this reason.
6. Cost and latency. None of the boards weigh tokens used or time taken, and those decide whether an agent is affordable.

A practical reading rule

Quote the board, the harness and the date with any score, and compare only rows from the same board. "Model X scores 97% on SWE-bench" with no board named is not a usable claim.

---

Local, open-weight models on SWE-bench

If your code cannot leave your machine, these are the open-weight numbers with a named source. Blank cells mean no published figure; we do not fill them with estimates.

Reading this table: open weights now reach the official Multilingual top five (GLM 5) and the vals.ai Verified top ten (Kimi K3, GLM 5.3, DeepSeek V4), but the models that do it need server hardware. The practical local coder on consumer hardware is Qwen3-Coder-Next at 70.6% Verified (the vendor's own figure), or the 7B-32B Qwen Coder models, which have no published SWE-bench score and should be judged on your own tasks. Setup: Ollama on Windows with Continue.dev; ranking of the local options in best local AI coding models.

---

How to test AI coding models yourself

Benchmarks narrow the shortlist; your repository decides. A five-step method that takes about a week of part-time effort:

Step 1: Define the job. Primary language, typical task (bug fix, feature, refactor, tests), codebase size, and whether the model will run as an agent or as a chat assistant.

Step 2: Build a small task set from your own history. Pull 5-10 closed issues or merged PRs, reset the repository to the commit before each fix, and keep the real tests as the oracle. That is SWE-bench's method applied to your code.

Step 3: Run every model the same way. Same prompt, same tool access, same turn or time budget, same context. Record whether the tests pass, how many turns it took, and how much you had to edit the result.

Step 4: Count cost and time. Tokens per task for API models; memory and tokens-per-second for local ones. A model that resolves 10% more tasks at three times the tokens may lose.

Step 5: Try it in the real workflow. Editor integration, git hygiene, how it behaves in CI. The benchmark winner is not always the one that fits.

A note on what counts as "good"

There is no benchmark threshold that translates into "safe to merge unreviewed". The official Verified board's best open scaffold resolves about four in five human-screened Python issues; the standardized Pro board's best resolves about three in five long-horizon tasks. Review everything.

---

Choosing a model from the benchmarks

For agentic fixing in Python repositories: start from the top of the vals.ai Verified board and Scale's Pro board. As of their dates, that means Claude Opus 5, DeepSeek V4 Pro, GPT-5.6 Sol and Grok 4.6 on Verified, and Muse Spark 1.1, GPT-5.4 and Claude Opus 4.6 on standardized Pro. Confirm with a pilot on your own issues.

For non-Python codebases: read the official Multilingual board. Gemini 3 Flash (72.7%), Claude 4.6 Opus (72.0%) and Claude 4.5 Opus (70.7%) lead it; GLM 5 (69.7%) is the open-weight option.

For front-end work with screenshots: Multimodal is the relevant board and nothing on it clears 36%. Expect to do more of the work yourself.

For code that must stay local: Qwen3-Coder-Next is the consumer-hardware option with a published Verified figure; GLM-5 if you have server hardware.

For a quick model-vs-model comparison: our best AI coding models guide and the LMArena coding leaderboard, which measures preference rather than test-passing and often disagrees with SWE-bench.

---

Frequently asked questions

What is the top SWE-bench Verified score in 2026?

It depends on the board. The official swebench.com board tops out at 79.2% (Sonar Foundation Agent + Claude 4.5 Opus, and live-SWE-agent + Claude 4.5 Opus, December 2025; no models after February 2026 have entries). The vals.ai board, updated September 1, 2026 with a bash-only harness, puts Claude Opus 5 at 97.0%, DeepSeek V4 Pro at 96.4% and GPT-5.6 Sol at 96.2%.

Where is the official SWE-bench leaderboard?

swebench.com. Its data is published in the site's repository, which is how the tables on this page were read. The site shows Verified, Lite, Full, Multimodal and Multilingual; SWE-bench Pro lives on Scale's SEAL leaderboard.

What was o3's SWE-bench Verified score?

69.1%, by OpenAI's figure at launch on April 16, 2025; o4-mini scored 68.1% in the same post.

What is the SWE-bench Pro leaderboard?

Scale AI's board for its 1,865-task long-horizon benchmark. On the standardized public set the top entries are Muse Spark 1.1 (61.5%), GPT-5.4 xHigh (59.1%) and Muse Spark (55.0%), as read on October 15, 2026. Vendor-reported Pro numbers, aggregated by llm-stats on October 14, 2026, run from 62.5% to 89.9% (Claude Opus 5.5) and are not comparable with the standardized board.

Why do coding agents list "SWE-bench Verified" scores that differ from this page?

Because every score is for a model inside a particular scaffold, and vendors run their own. The same model lands several points apart between the official self-submission board, vals.ai's fixed harness, and a vendor's tuned agent. Always ask which board and which date.

How does SWE-bench differ from HumanEval?

HumanEval is 164 standalone Python functions written from a docstring; SWE-bench is real repository issues that need navigation, a multi-file fix and a passing test suite. HumanEval is saturated for frontier models; SWE-bench Pro and Multilingual are not.

How often should I re-check the boards?

The official boards update when submissions land (Verified's last entry is February 26, 2026); vals.ai has stopped adding models; Scale's Pro board updates irregularly. Checking quarterly is enough for most teams; re-run your own task set when you consider switching.

---

Conclusion: benchmarks as a starting point, not a verdict

What the SWE-bench family tells you, as of the dates above:

• The original 2023 task set went from 1.96% (Claude 2) to 52.62% on the official Full board.
• Verified reads 79.2% on the official board (February 2026 systems) and 97.0% on vals.ai (September 2026 models); vals.ai calls it saturated.
• SWE-bench Pro is where models still separate: 61.5% at the top of Scale's standardized board, with vendor-reported figures up to 89.9% that cannot be compared with it.
• Multilingual (72.7% top) and Multimodal (35.98% top) are the boards to read if your work is not Python text.
• Open weights have reached the top five of an official board (GLM 5 on Multilingual) and the top ten on vals.ai, but at server-class sizes; the consumer-hardware option with a published figure is Qwen3-Coder-Next at 70.6% Verified.

Use the boards to shortlist, then run the candidates on five to ten of your own closed issues with the real tests as the judge. That test is the only benchmark that measures your codebase.

Ready to pick a model? Start with the best AI coding models guide, then test the top candidates on real work.