Isometric task assembly line: a document, a container, a solution and a test rack joined on one base

For AI labs

What gets a benchmark task accepted

Terminal-Bench 2.0 kept 89 of 229 submitted tasks. The accepted ones share four parts: a real job, a pinned environment, an expert solution and tests that fail every shortcut. Here is each one, and where submissions fall down.

Prashant Kumar Singh · 6 November 2026 · 4 min read

When the Terminal-Bench team published the 2.0 paper in January 2026, they included a number most benchmark papers leave out: how many submitted tasks they threw away. Ninety-three contributors submitted 229 tasks. Eighty-nine made the final set, about 39%, after roughly three reviewer-hours per accepted task.

That ratio matches what I see reviewing tasks for data partners. Most rejected tasks are not bad ideas. They are missing one of four parts, and the part most often missing is the one that decides whether the task measures anything: the tests.

1. A real job, not a puzzle

An accepted task starts with work someone would actually be asked to do. "This nightly report is wrong, fix it." "Migrate this config to the new format without breaking the deploy." "Make these tests pass without changing their assertions." The realism matters for two reasons. It is what the model will meet in production, and it is what makes the task hard in a fair way: the difficulty comes from the problem, not from a trick in the wording.

Puzzles get rejected because they measure something narrow. A task about counting letters in a sentence tests tokenisation, not engineering. Reviewers look for tasks where an experienced engineer would need to read, reason and verify, and where the final state can be checked.

2. A pinned environment

Every accepted Terminal-Bench task runs inside a Docker image with versions fixed. Same base image, same packages, same data, every run. Without that, a task passes on one machine and fails on another, and nobody can tell whether the model or the environment is at fault.

Pin the base image to a digest or a specific tag, never latest.
Pin every package the solution or the tests import.
Copy data into the image. Do not download it at run time; network luck is not part of the task.
Record the exact command the grader runs, so a reviewer can reproduce the result without reading your code.

3. An expert solution, run fresh

A reference solution proves two things at once: the task is solvable, and the grader recognises a correct answer. Reviewers run it before anything else. If it fails, the task is broken, not hard. Anthropic's guide to agent evals makes the same point: a 0% pass rate across a hundred attempts almost always means a task or grader bug, not a genuinely impossible problem.

The failure I see most often here is a solution that passed once, weeks ago, on the author's machine. Then the environment changed, or a test was tightened, and nobody re-ran it. I re-run the reference solution in a clean container immediately before every submission. It takes two minutes and catches a surprising number of problems.

4. Hidden tests that fail every shortcut

This is where tasks are won and lost. Tests check the final state of the environment, not the model's description of what it did. They are hidden from the agent. And they are written to fail the three common shortcuts: hard-coding the expected output, fixing one of two bugs, and doing nothing at all.

A good exercise is to try to cheat your own task. Run an agent that makes no changes. Does it fail? Paste the expected output straight into the result file. Does it fail? Fix half the bug. Does it fail? If any of those pass, the task is not ready, no matter how realistic the scenario is.

Slide: a real job, realistic work not a puzzle — From the carousel: the four parts of an accepted task, and where submissions fail.

Slide: a pinned sandbox, same environment on every run — From the carousel: the four parts of an accepted task, and where submissions fail.

Where tasks actually fail

Reviewers spent about three hours per accepted task in Terminal-Bench 2.0. In my own reviews, test gaps are the most common fix by a wide margin, followed by under-specification: the task says "produce a report" but the grader checks a specific filename, column order or rounding that the instructions never mention. The model fails for a reason that has nothing to do with skill, and the task gets rejected for being unfair.

None of this is secret. The Terminal-Bench paper, the Berkeley trustworthy-benchmarks checklist and Anthropic's evals guide all say versions of the same thing. The difference between a 39% acceptance rate and a 90% one is whether somebody runs those checks on every task before it leaves the building.

If you're a founder

The same four parts make a good acceptance test for a feature: a real user job, a fixed environment to test in, a known-good example, and a check that fails when the feature only half works. If your QA is "it looked right on my laptop", it is missing three of them.

If you run an AI lab or data team

If your pipeline's acceptance rate is well under Terminal-Bench's 39%, the fix is almost never more contributors. It is a pre-review gate that runs the reference solution and the shortcut checks before a human reviewer sees the task.

Sources