The five checks behind 45 accepted tasks

Acceptance rates in task pipelines are low. Terminal-Bench 2.0 kept 39% of submissions. OpenAI's original SWE-bench Verified effort filtered out 68% of the samples it screened. When I started writing tasks for data platforms, my first few went through the same grinder. What changed the rate was not writing cleverer tasks. It was a fixed routine I run before submitting anything.

Here are the five checks. Each one takes a few minutes. Together they are the difference between a task a reviewer sends back and a task a reviewer approves.

Check 1: the expert solution passes, run fresh

I build the environment from scratch, apply my reference solution, and run the grader. It must pass. Not "passed last week", not "passed on my laptop". Fresh container, today. Anthropic's evals guide puts it bluntly: if a task has a 0% pass rate across a hundred attempts, the task is almost certainly broken, not hard. The reference solution is how you prove it is not broken.

This catches drifting dependencies, a test that was tightened after the solution was written, and data that changed shape. It is the single most common reason a task that looks fine on paper gets rejected.

Check 2: an empty agent fails

I run an agent that does nothing, then grade. The score must be zero. Berkeley's trustworthy-benchmarks checklist calls this the null-agent baseline, and it is the floor every task should clear. If a do-nothing run scores anything above zero, the grader is rewarding the starting state, and the model will learn to do as little as possible.

Check 3: a partial fix fails

Most realistic tasks have more than one thing wrong. A report has a sign error and a timezone error. A migration breaks two call sites. I fix exactly one and grade. The score must still be zero, or the task must be explicitly designed for partial credit with the rubric saying so. A task that gives full marks for half the work teaches the model that half the work is enough.

Check 4: hard-coding fails

I write the expected output directly into the result file without running any logic, then grade. It must fail. The way to make this work is hidden inputs: the grader runs the solution against data the agent never saw, so a pasted answer has nothing to match. If your expected output is sitting in the environment where the agent can read it, this check will fail and so will the task in review.

Check 5: a different correct solution passes

I solve the task a second way. Different approach, different code, same correct result. The grader must pass it. Tests that only accept my reference implementation are testing style, not correctness, and they will reject valid solutions from models and from the human reviewers who try the task. Grade the outcome, never the path.

What the routine looks like

The whole routine is scripted. A task does not leave my machine until all five lines are green. When a task comes back from review anyway, the reason goes into the script as a sixth check, which is how the list grew from three to five.

Why it works

Reviewers are doing exactly these checks, by hand, at about three hours per task. If the task arrives already passing them, the reviewer's job becomes confirming rather than debugging, and confirmation is fast and usually positive. A 90% acceptance rate is mostly a matter of moving the review to before submission.