For AI labs
Write task specs nobody can misread
When OpenAI built SWE-bench Verified, 93 developers screened 1,699 samples and filtered out 68%, many for under-specified problem statements or unfair tests. A vague spec is the cheapest bug to fix and the most expensive one to ship. Five rules for writing specs that authors and models read the same way.
Prashant Kumar Singh · · 4 min read
In short
- OpenAI's SWE-bench Verified screen removed 68% of the samples it checked; under-specified problem statements and over-specific tests were leading reasons.
- A spec states the end state, lists what is out of scope, gives one worked example, defines "correct" in terms a test can check, and is piloted on ten tasks before scale.
- Ambiguity is cheap to fix in the spec and expensive to fix after fifty tasks have been written against it.
SWE-bench Verified exists because the original benchmark had problems that were not the model's fault. OpenAI had 93 software developers screen 1,699 samples. After screening, 68% were filtered out. The two most common reasons were problem statements that did not say enough for a correct fix to be written, and tests so specific that a correct fix written another way would fail.
Both are spec failures, and both are cheap to prevent. Here are the five rules I use when writing the specification for a task programme, whether the tasks are for a lab's benchmark or for a training set.
1. State the end state
Say what must be true when the task is done, in terms of files, outputs and behaviour. "Fix the date bug" is a direction. "After the change, parse_date('2026-02-30') raises ValueError and the existing test suite passes" is an end state. Authors can write to the second; models can be graded on it.
2. List what is out of scope
Authors fill silence with guesses. If the spec does not say whether tasks may use the network, whether a second valid solution should pass, or whether the agent may read the test files, each author decides for themselves and you get five different conventions in one batch. Write the exclusions down, even the obvious ones.
3. Give one worked example
A full task that passed review, with a note on why. The example answers a hundred questions the spec never thought to ask: how long the instruction should be, how the tests are laid out, how the reference solution is documented. One good example does more than three pages of rules.
4. Define "correct" in terms a test can check
If a requirement cannot be checked by a test or a rubric criterion, it is not a requirement, it is a hope. "The code should be clean" is a hope. "No function longer than 50 lines; ruff passes with the project config" can be checked. Over-specific tests are the opposite failure: they check one implementation instead of the behaviour. The rule is to check the outcome, never the path.
5. Pilot the spec on ten tasks
The first ten tasks written against a spec surface most of its ambiguities. Read them side by side, note where authors diverged, and fix the spec before anyone writes task eleven. This is the step most programmes skip, and it is why batch three has the same problems as batch one.
What this looks like in a real spec
| Section | What it contains | Length |
|---|---|---|
| End state | What must be true when the task is complete | 3–6 lines |
| Out of scope | Explicit exclusions, including the obvious ones | A list |
| Worked example | One accepted task with a note on why it passed | One task |
| Definition of correct | Checks a test or rubric can run; outcome, not path | A list |
| Pilot | Ten tasks, read side by side, spec revised | One week |
The spec that comes out of this is usually shorter than the one it replaces, and the acceptance rate on the next batch is usually the proof.
Questions people ask
- Why were so many SWE-bench samples filtered out of SWE-bench Verified?
- OpenAI's screen of 1,699 samples by 93 developers filtered out 68%. Leading reasons were under-specified problem statements, tests that would fail a correct fix written differently, and environment problems.
- What makes a benchmark task under-specified?
- The problem statement does not contain enough information for a competent engineer to write a correct fix without guessing: missing end state, unstated constraints, or requirements that no test can check.
- How do you test a task specification before scaling?
- Pilot it on ten tasks, read them side by side, note where authors made different choices, and revise the spec before more tasks are written.
If you're a founder
A feature brief has the same failure mode as a task spec. If two engineers read it and build different things, the brief was the bug. The one-page scope from the previous post is the fix.
If you run an AI lab or data team
Send me your task spec or guideline and I will return a written review within 48 hours: where authors will diverge, what cannot be tested as written, and what to add before the next batch.
Sources