For AI labs
Read the trajectory, not the pass rate
A pass rate tells you how often a model solves a task. A trajectory tells you why it did not. I read every failed run before a task ships, because the failure cause decides the fix, and three runs failing the same way is a task bug, not a hard task.
Prashant Kumar Singh · · 4 min read
In short
- Anthropic's evals guide: a 0% pass rate across many attempts is almost always a broken task, not a hard one. Only reading the runs tells you which.
- Four failure causes need four different fixes: misread spec (tighten the instruction), missed hidden input (task is fine), fixed the sample only (strengthen tests), tool error (fix the environment).
- Three runs failing the same way is a task bug. Three runs failing different ways is a hard task.
The number everyone reports for a task is its pass rate: run a model ten times, count the successes. It is a useful number and it answers one question only, how often. The question that decides whether the task is any good is why, and the only place that answer lives is in the trajectories, the step-by-step record of what the agent did.
I read every failed run before a task ships. Not a sample. All of them. Here is what I am looking for and what each finding changes.
The rule of thumb from Anthropic's guide
Anthropic's engineering guide on agent evals makes a point that experienced task authors learn the hard way: a task with a 0% pass rate across many attempts is almost always broken rather than hard. The guide's example is a CORE-Bench score that went from 42% to 95% after grading bugs were fixed and the scaffold changed. The models did not improve; the task stopped lying. A pass rate cannot tell the difference between hard and broken. A trajectory can.
Four causes, four fixes
| What the trajectory shows | Diagnosis | Fix |
|---|---|---|
| Agent solves a different problem from the one intended | Misread spec | Tighten the instruction; add the missing constraint |
| Agent solves the sample and stops | Hidden input missed | Task is fine; this is the capability being measured |
| Agent special-cases the visible test inputs | Weak tests | Hidden inputs at grading time; property tests |
| Agent gives up after a tool or environment error | Environment bug | Fix the container, pin the dependency, raise the limit |
Each row is a different task edit, and some rows mean no edit at all. A pass rate of 30% could be any mix of the four. Reading the runs tells you the mix.
Group by cause across runs
The useful unit is not a single failed run but the set of them. Group failures by cause. Three runs failing the same way, on the same step, for the same reason, is a task bug: something in the spec, the tests or the environment is pushing every attempt into the same hole. Three runs failing three different ways is a hard task: the model is reaching different points and running out of capability at each. The first needs fixing. The second is what a benchmark is for.
Read the successes too
A passing run can be a failure in disguise. If the agent reached the answer by a route the task did not intend, by finding the expected output in a cache, by guessing a value that happened to match, by exploiting a harness detail, the pass is a hack you have not named yet. One in every few successful trajectories deserves the same reading as the failures.
How long this takes
Ten runs on a task take an hour to read properly. That hour is cheaper than the three reviewer-hours a rejected task costs, and far cheaper than a broken task that ships and spends months in a benchmark measuring nothing. It is also the most educational hour in task authoring: the trajectories show you how models actually read your instructions, which is never quite how you meant them.
Questions people ask
- What is an agent trajectory?
- The step-by-step record of what an agent did on a task: the commands it ran, the files it read and changed, its reasoning, and where it stopped. It is the only record that explains why a run failed.
- Does a 0% pass rate mean a task is too hard?
- Usually not. Anthropic's evals guide notes a 0% rate across many attempts is almost always a broken task. Reading the trajectories shows whether the failures share one cause (a task bug) or vary (a hard task).
- How do you tell a hard task from a broken one?
- Group failed runs by cause. Many runs failing the same way at the same step points to a spec, test or environment bug. Runs failing in different ways at different points indicate the model is running out of capability, which is what a benchmark measures.
If you're a founder
Your AI feature's logs are its trajectories. Reading the ones where users abandoned mid-flow tells you what the metric cannot: where the feature lost them and why.
If you run an AI lab or data team
Send one task and its runs. The free review reads them, groups the failures, and tells you which rows of the table above apply.
Sources