For AI labs
2026 in agent benchmarks: what changed
Verified task sets, harness exploits found and fixed, real-work tasks written by professionals, and the end of the pass rate as proof. A short year in review for anyone who writes, reviews or buys evaluation tasks, with the four changes that will still matter in 2027.
Prashant Kumar Singh · · 4 min read
In short
- Terminal-Bench 2.0 showed the filter: 89 of 229 submitted tasks kept, after roughly three reviewer-hours each.
- Berkeley RDI showed harness exploits scoring 100% with zero tasks solved, and published the checklist that stops them.
- GDPval set the bar for who writes tasks: professionals with years of experience, multiple review rounds, expert graders. Pass rates stopped being proof; trajectories and graders are.
A year ago the question about an agent benchmark was "what did the model score". Today the first question is "can the score be trusted", and 2026 is the year that question got real answers. Four changes, each with the public work behind it, and what each one means for the people who write tasks for a living.
1. Task sets got verified
Terminal-Bench 2.0's paper, published in January, documented what a serious filter looks like: 229 tasks submitted, 89 accepted, roughly three reviewer-hours per task, every accepted task verified across multiple runs in a pinned environment. The 39% acceptance rate was not a scandal; it was the first time a benchmark published the number. SWE-bench Verified had shown the same thing from the other direction, filtering 68% of what it screened. The lesson settled: a task set that has not been through human verification is mostly noise, and everyone now knows it.
2. Harness exploits were found and named
In April, Berkeley's RDI group published work showing that an agent could score 100% on well-known benchmarks without solving a single task, by exploiting the harness: reading expected outputs, signalling success directly, modifying the grader. They published a checklist: isolate the grader, run the null agent, lock the test files, check that success cannot be faked. Every task programme that did not already run those checks started to, and task authoring acquired a security mindset it had lacked.
3. Real-work tasks set the bar for authors
GDPval, from late 2025, became the reference point through 2026 for what a task should be: based on a real work product, written by someone with years of experience doing that work, reviewed about five times, graded blind by expert peers against a human deliverable. PaperBench had done the same for research reproduction with its 8,316-leaf rubric trees. The effect on the market was visible: generalist annotation pools shrank, specialist teams grew, and "who wrote this task" became a question clients ask.
4. Pass rates stopped being proof
Anthropic's engineering guide in January said what experienced authors knew: a 0% pass rate is usually a broken task, and a score moves dramatically when grading bugs are fixed. Judge's Verdict put a number on human grader agreement, around kappa 0.80, and held model judges to it. By year end the serious conversation about a benchmark result was about trajectories, grader agreement and harness integrity, with the pass rate as the output of those rather than evidence on its own.
| Change | Source | What it means for task authors |
|---|---|---|
| Verified task sets | Terminal-Bench 2.0, SWE-bench Verified | Run the reviewer's checks before the reviewer does |
| Harness exploits named | Berkeley RDI checklist | Isolate the grader, null agent, locked tests, every task |
| Real-work bar | GDPval, PaperBench | Domain experts, multi-round review, expert graders |
| Pass rate demoted | Anthropic evals guide, Judge's Verdict | Read trajectories, measure grader agreement |
What to expect in 2027
More of the same, further along. Verification will be expected rather than notable. Checklists like Berkeley's will be built into harnesses by default. Real-work evaluations will spread to more domains, and the scarce resource will be qualified authors and graders, not compute. For anyone who writes or buys tasks, the practical consequence is that the pre-submission routine, reference re-run, hidden tests, null agent, second solution, trajectories read, is no longer a differentiator. It is the entry requirement.
Questions people ask
- What changed in AI agent benchmarks in 2026?
- Four things: task sets were verified by humans (Terminal-Bench 2.0 kept 89 of 229 submissions), harness exploits were documented and a checklist published (Berkeley RDI), real-work tasks by professionals became the bar (GDPval, PaperBench), and pass rates were demoted in favour of trajectories and grader agreement.
- What is a harness exploit in AI benchmarks?
- A way for an agent to score on a benchmark without solving tasks, such as reading expected outputs, signalling success directly, or modifying the grader. Berkeley RDI showed such exploits reaching 100% and published checks to prevent them.
- What will matter in AI evaluation in 2027?
- Verification as standard, exploit checks built into harnesses, real-work evaluations across more domains, and qualified task authors and graders as the scarce resource.
If you're a founder
If your product claims a benchmark score, expect customers in 2027 to ask how it was verified. The same four questions apply to your own evaluation.
If you run an AI lab or data team
Want your 2027 task programme to start at the entry requirement? Send one task and the free review applies all four changes to it.
Sources