Five hidden-test patterns that can't be gamed

In April 2026 Berkeley's RDI group published a checklist for trustworthy agentic benchmarks after finding harness exploits that let an agent score 100% without solving a single task. The checklist covers the harness: isolate the grader, run the null agent, lock the test files, check that success cannot be signalled directly. Every task programme should run it.

The harness is necessary and not sufficient. A perfectly isolated grader running a weak test still passes a shortcut. These are the five patterns I use for the tests themselves, in the order I apply them.

1. Hidden inputs, generated at grading time

The agent sees the specification and the sample data. The grader runs on inputs it generates or mounts at grading time, drawn from the same distribution but never present in the task directory. Special-casing the sample becomes useless because the sample is not what is graded. For data tasks this means a different file; for code tasks, a different set of arguments; for agentic tasks, a different starting state.

2. Final state, not transcript

Grade what is true at the end: the files on disk, the rows in the table, the output of running the program. Never grade the agent's own account of what it did, and never grade a log line it could have printed. The model that writes "All tests passed" to stdout should score exactly the same as the model that writes nothing.

3. Property tests

Instead of asserting one expected output, assert a property that must hold for any input: the function is idempotent, the total is preserved, the sort is stable, the parser round-trips. Then run it on a few hundred random inputs. A property test catches the solution that works on the three cases the author thought of and nothing else, which is the most common shape of a wrong answer.

4. A second valid solution must pass

Before a task ships, write a second correct solution that takes a different approach, and confirm the tests pass it. SWE-bench Verified filtered out tasks whose tests only accepted one implementation; those tests graded the path instead of the outcome and failed correct answers. If a second solution fails your tests, the tests are over-specific and the task will be rejected at review or, worse, accepted and then wrong.

5. The grader runs out of reach

This one is from the Berkeley list, and it belongs here because test patterns are worthless without it. The grader runs in a separate container, the test files are mounted read-only after the agent's turn ends, and the result is written somewhere the agent cannot touch. The agent's environment should contain nothing that, if modified, changes the score.

Why this is worth the time

A task takes hours to write. The tests take a fraction of that and decide whether the hours measured anything. Terminal-Bench 2.0 accepted about 39% of submitted tasks after roughly three reviewer-hours each; most of what was rejected failed on exactly the problems these patterns prevent. Writing the tests this way is how a task clears review the first time.