Notes
🔍

SWE-bench

Most coding benchmarks ask for a short function that passes a few unit tests, and models now sit near the ceiling of them. SWE-bench asks something harder and much closer to the actual job: here is a real repository and a real GitHub issue — produce a patch that makes the project's own tests pass.

How a task is built
Each instance is mined from the history of a popular open-source Python project (Django, scikit-learn, sympy and others). The pipeline finds a merged pull request that closed an issue and reconstructs the moment before the fix:

  • The repository at the parent commit, plus the issue text — that is all the agent sees.
  • FAIL_TO_PASS tests taken from the real pull request, which must go from failing to passing.
  • PASS_TO_PASS tests, which must not break — the guard against a patch that scores by deleting things.

Grading is by execution, not similarity: the patch is applied inside a container and the tests are run. There is no partial credit, and the agent has to locate the right files itself across thousands.

Reading the number
Resolved percentage on SWE-bench has become the headline metric for coding agents, so it matters which set a figure refers to. SWE-bench Lite is a 300-instance subset for cheap iteration; SWE-bench Verified is 500 instances human-screened for solvability and clear issue text, and is what vendors now quote; multimodal and multilingual variants extend past Python. Scores are not comparable across them, and because the tasks come from public repositories that models have trained on, contamination remains a live concern.

SWE-bench evaluates agents on real software-engineering work: given a repository snapshot and a GitHub issue, produce a patch that makes the project's own tests pass. Grading is by execution — designated failing tests must pass and previously passing ones must not break — so there is no partial credit and the agent must find the relevant files itself. Its Verified and Lite subsets are the figures commonly quoted for coding agents, and are not interchangeable.

A written exam about cooking tells you very little about whether someone can run a kitchen. SWE-bench is the practical test instead of the exam.

It takes real bug reports from real software projects, hands them to an AI along with the actual code, and asks for a fix. Then it does the only thing that really counts: it runs the project's own tests. The fix either makes them pass or it does not.

That is why these scores get quoted so often — they come from genuine work rather than puzzles. The catch is that there are several versions of the test, some easier than others, so two numbers are only worth comparing when they came from the same one.