For AI labs
What "verified" means in SWE-bench Verified
OpenAI's screen of the original SWE-bench removed 68% of the samples it checked. The three reasons they failed, under-specified issues, over-specific tests and broken environments, are a checklist for anyone writing repository tasks. Here it is, with the rule I use for each.
Prashant Kumar Singh · · 4 min read
In short
- 93 developers screened 1,699 SWE-bench samples; 68% were filtered out of what became SWE-bench Verified.
- Three failure types: issue text too vague to fix from, tests that only accept one implementation, and environments where even the gold patch did not run cleanly.
- The rule before a repository task ships: a senior engineer could fix it from the issue alone, the tests accept any correct fix, and the gold patch passes in a fresh container.
SWE-bench became the standard way to measure coding agents on real repository work: here is a GitHub issue, here is the repository, produce a patch that makes the hidden tests pass. It was also, in its original form, partly broken. Not in a way that was obvious from the leaderboard, which is the point of this post.
OpenAI's response was to screen it. Ninety-three software developers reviewed 1,699 samples, and 68% of those were filtered out. What remained became SWE-bench Verified. The reasons for filtering are the most useful public document on what makes a repository task fair, so here they are, each with the authoring rule it implies.
Failure 1: under-specified issues
Many issue descriptions did not contain enough information for a correct fix to be written. The expected behaviour was implied, the failing case was not stated, or the issue described a symptom whose cause was somewhere else entirely. A model that failed these was not failing at engineering; it was failing to guess what the author meant. Those samples measured ambiguity, not capability.
The rule: before the task ships, give the issue text alone to a senior engineer who has not seen the gold patch. If they cannot say what a correct fix must do, the issue text is not finished.
Failure 2: over-specific tests
Some test suites would pass only the exact gold patch. A correct fix written differently, a different function name, a different but equivalent control flow, a fix one level up the call stack, failed. These tests graded the path, not the behaviour, and a model that solved the problem could still score zero.
The rule: write a second correct fix that takes a different approach, and confirm the tests pass it. If they do not, the tests are checking an implementation rather than an outcome, and they must change.
Failure 3: environment problems
In some samples the tests did not run cleanly even with the gold patch applied: a dependency had moved, a build step was missing, a test depended on the network or on a timezone. The task was unsolvable regardless of the model, and nothing on the leaderboard said so.
The rule: the gold patch passes in a fresh container, from the pinned image, ten times. Not once on the author's machine. If it does not, the environment is the bug.
| Failure | What it measured | Rule |
|---|---|---|
| Under-specified issue | The model's guess at the author's intent | A senior engineer can fix it from the issue alone |
| Over-specific tests | Whether the fix matched the gold patch | A second, different correct fix passes |
| Broken environment | Nothing | Gold patch passes ten times in a fresh container |
Why 68% is not a scandal
A filter that removes most of what it checks sounds alarming, and it is actually the normal rate for any benchmark that has not been through human verification. Terminal-Bench 2.0 accepted about 39% of submissions with reviewers in the loop from the start. The difference between a 32% survival rate and a 90% acceptance rate is not talent. It is running these three checks before review instead of letting review run them for you.
Questions people ask
- What is SWE-bench Verified?
- A human-validated subset of the SWE-bench coding benchmark. OpenAI had 93 developers screen 1,699 samples and filtered out 68%, keeping only tasks with clear issue text, fair tests and working environments.
- Why were SWE-bench samples filtered out?
- Three main reasons: issue descriptions too vague to write a correct fix from, tests that only accepted the exact gold patch, and environments where tests did not run cleanly even with the correct fix applied.
- How do you check a repository task is fair before submitting it?
- Have a senior engineer fix it from the issue text alone, write a second correct fix with a different approach and confirm the tests pass it, and run the gold patch ten times in a fresh container from the pinned image.
If you're a founder
A product spec has the same three failures: too vague to build from, acceptance tests that only pass one design, and an environment nobody has run it in. The one-page brief is the fix.
If you run an AI lab or data team
Writing repository tasks? Send one and the free review runs all three checks and tells you which it would have failed at screening.
Sources