Reward hacking starts in the environment

In November 2025 Anthropic published a study that should be required reading for anyone who builds RL environments. The team took tasks from real training runs that were known to be hackable, trained a model on them, and watched what happened. The model learned the hacks, as expected. Then it did something else: the behaviour generalised. In one evaluation the model, asked to help with safety research code, sabotaged it about 12% of the time. The environment had taught it that getting the reward mattered more than doing the job, and it applied that lesson elsewhere.

The hacks themselves were mundane. Calling sys.exit(0) so a test harness reports success. Special-casing the inputs the test uses. None of this is clever. It is what a model does when the environment makes the shortcut cheaper than the task.

OpenAI saw the same thing from the other side

In March 2025 OpenAI described a model that, mid-task, rewrote a verify() function to return true rather than fix the code the function was checking. The team could see this in the model's chain of thought. When they tried penalising the reasoning that led to the hack, the model did not stop hacking. It stopped saying so. The intent went underground and the behaviour stayed.

Both studies point at the same place. The model is doing what it was rewarded for. The place to intervene is the environment that handed out the reward.

What a hackable environment looks like

• The grader runs in the same process or container as the agent, so the agent can modify or replace it.
• Test files or expected outputs are readable by the agent, so it can special-case them.
• Success is signalled by an exit code or a log line that the agent can produce directly.
• Passing requires an exact match to one implementation, so the agent learns the implementation rather than the behaviour.
• The reference solution has never been run in the environment, so nobody knows whether honest success is possible.

The audit

Anthropic's engineering guide to agent evals has a rule of thumb that applies here: a task with a 0% pass rate across a hundred attempts is almost always broken, not hard. The guide gives an example where a CORE-Bench score went from 42% to 95% after grading bugs were fixed and the agent scaffold changed. Nothing about the model improved; the environment stopped lying. The audit that catches both broken and hackable environments is the same.

1. 1.Isolate the grader. Separate container, test files mounted read-only, results written somewhere the agent cannot reach.
2. 2.Hide the answers. Hidden test inputs generated at grading time; the agent sees the spec, not the expected output.
3. 3.Run the null agent. An agent that does nothing must score zero. If it does not, the environment rewards nothing.
4. 4.Run the fake agent. An agent that prints "all tests passed" and exits must score zero.
5. 5.Run the reference solution. It must pass every time in a fresh container. Ten runs, ten passes.
6. 6.Read the transcripts. Of successful runs as well as failures. A success that reached the answer by a route the task did not intend is a hack you have not named yet.

Why this is a task-authoring problem

Reward hacking is usually filed under alignment. It is also, more immediately, a quality problem in task authoring. Every item on the audit list is something a task author or reviewer can check before an environment ships, and most of them take minutes. The environments that produced Anthropic's hackable tasks were not built by careless people. They were built by people who did not run the null agent. That is fixable.