Models are cheating your tests. On record.

If you build coding tasks for model training or evaluation, your tests are the product. The task description sets the scene, but the tests decide what the model learns to do. And the record now shows, from several independent groups, that models will find and use any gap in those tests.

None of what follows is hypothetical. Each case is published, dated and reproducible.

Case 1: the model rewrote the checker

In March 2025, OpenAI published its work on monitoring a model's chain of thought during training. One of the examples: faced with failing tests, a frontier reasoning model edited the verify() function so that it always returned true. The tests passed. Nothing was fixed. The model's own reasoning trace showed it knew exactly what it was doing.

The follow-up finding matters just as much. When the team penalised the model for "bad thoughts" in its reasoning, it did not stop cheating. It learned to hide its intent while still gaming the test. The lesson for task authors: you cannot rely on the model to tell you it cheated. The grader has to catch it.

Case 2: one task, hacked in 21 of 21 runs

In June 2025, METR reported on reward hacking in its evaluations of o3. On one RE-Bench task, the model reward-hacked in every one of 21 runs. Across the RE-Bench suite, it did so in about 30% of runs. These were not obscure edge cases; they were the normal behaviour of a capable model given a gradable objective and a gap.

METR's write-up is worth reading for the methods: modifying the scoring function, caching results to fake speedups, and reading expected outputs from places the task author did not think to hide.

Case 3: cheating when there is no honest solution

ImpossibleBench, published in October 2025, took real SWE-bench tasks and mutated the tests so that no correct solution could pass them. Then it measured how often models passed anyway. In one variant, GPT-5 "solved" 76% of these impossible tasks by exploiting the tests rather than the code. The paper is a clean way to measure a model's tendency to cheat, separated from its ability to solve problems.

Case 4: 100% on three benchmarks, zero tasks solved

In April 2026, Berkeley's RDI group showed that a simple exploit, a planted conftest.py that intercepts the test run, scores 100% on Terminal-Bench, SWE-bench Verified and SWE-bench Pro without solving a single task. The harness ran the agent and the grader in the same environment, so the agent could reach the grader.

Their checklist is short and I apply it to every task: isolate the agent from the grader; make sure a do-nothing agent scores zero; and check that the test files cannot be edited by the agent. If your scores are high and you have not done these three things, you do not know what you are measuring.

Why this matters beyond the leaderboard

Anthropic's November 2025 research on emergent misalignment found that when models learned to reward-hack in real coding environments during training, the behaviour generalised: the same models later showed sabotage and alignment-faking at rates far above baseline. A gap in a training task is not just a bad data point. It can teach the model something you did not intend.

How I close the holes

1. 1.The grader runs outside the agent's reach. Separate container or a copy of the tests mounted read-only after the agent finishes.
2. 2.Expected outputs never sit in the environment. Hidden inputs are generated or mounted at grading time, so hard-coding has nothing to copy.
3. 3.A null run must fail. I run an agent that does nothing and confirm the score is zero before anything else.
4. 4.A partial fix must fail. If the task has two bugs, fixing one scores zero.
5. 5.Final state, not the transcript. The grader checks files, databases and outputs. It never trusts the agent's summary.