For AI labs
Hard but fair: how I calibrate task difficulty
Too easy teaches nothing. Too hard for the wrong reason teaches the wrong thing. The target is a task a strong engineer solves and a frontier model solves sometimes, and the only way to know you have hit it is to run real models and read why they failed.
Prashant Kumar Singh · · 4 min read
In short
- The target difficulty: a strong engineer solves it; a frontier model solves it sometimes. Every model passing means too easy; no model passing means check the task before calling it hard.
- Hard for the right reason: the model misses a real engineering step. Hard for the wrong reason: a vague spec, a flaky environment, a trick.
- Fair difficulty levers: more interacting components, a second bug, a stricter output contract. Unfair levers: ambiguity, hidden requirements, gotchas.
Difficulty is the least discussed part of task authoring and the one that decides whether a benchmark is worth anything. A task every model passes is a unit test. A task no model passes is either a frontier or a bug, and the leaderboard cannot tell you which. Here is how I set difficulty and how I check that it landed where I meant.
The target
A task a strong engineer in the domain solves in a sitting, and a frontier model solves sometimes. "Sometimes" is the useful region: a pass rate that is neither zero nor one carries information about the model, and a pass rate that moves between model versions is what a benchmark exists to produce. If every model you run passes every time, raise the difficulty. If none passes ever, do not raise anything yet. Check the task.
Why 0% is a warning, not a badge
Anthropic's evals guide puts it plainly: a 0% pass rate across many attempts usually means a broken task rather than an impossible one. The guide's example, a benchmark score that moved from 42% to 95% after grading bugs and scaffold problems were fixed, shows how much of apparent difficulty is accident. Before a task is called hard, the failed trajectories have to be read, and the failure cause has to be a real engineering step the model missed, not a spec it could not parse or a container that ran out of memory.
Hard for the right reason, hard for the wrong reason
| Right reason (keep) | Wrong reason (fix) |
|---|---|
| The model misses a step a senior engineer would take | The spec leaves two valid readings |
| The bug is silent and needs reasoning about numerics | The environment drifts or flakes |
| Two components interact in a non-obvious way | A requirement exists only in the author's head |
| The output contract is strict and stated | The test accepts only one implementation |
| The task needs several tools used together | A trick: a misleading comment, a decoy file |
The right-hand column makes a task harder and teaches the model nothing, because the model that fails it has not failed at engineering. Worse, a model that passes it has learned to read the author's mind, which is not a capability anyone wants to train.
The levers that keep it fair
- More interacting components. The bug is in how two modules talk, not inside one of them.
- A second bug. Fixing the obvious one exposes the next; the task is done only when both are.
- A stricter output contract. An exact schema, a tolerance, a performance bound, all stated in the spec.
- Larger scope. The fix touches three files instead of one, with a test that checks them together.
- Realistic noise. A codebase with the usual cruft, not a toy, so finding the relevant part is part of the task.
The process
- 1.Write the task and the reference solution. Run the reference ten times in a fresh container.
- 2.Run two or three real models, several attempts each. Record pass rates.
- 3.Read every failed trajectory. Group by cause. Fix anything in the wrong-reason column.
- 4.If pass rates are all 100%, pull a fair lever and go back to step 1. If all 0%, look harder at step 3.
- 5.Ship when the pass rate is in the informative region and every failure you read was a real step missed.
What difficulty is for
A benchmark exists to separate models that will be built next year from the ones that exist now. That requires tasks in the band where today's models sometimes fail and tomorrow's will sometimes pass. Setting that band is a craft, and the tools are the ones above: run real models, read the runs, fix the wrong reasons, keep the right ones.
Questions people ask
- How hard should a benchmark task be?
- Hard enough that a frontier model solves it only sometimes, and never so hard that no model solves it without first checking that the task itself works. A strong engineer in the domain should be able to solve it in a sitting.
- What makes a task hard for the wrong reason?
- Ambiguous specs, flaky environments, requirements that exist only in the author's head, tests that accept one implementation, and tricks such as misleading comments. These make tasks harder without measuring any engineering capability.
- How do you make a task harder fairly?
- Add interacting components, a second bug that appears after the first is fixed, a stricter stated output contract, a larger scope across files, or realistic codebase noise. All of these raise difficulty through real engineering steps.
If you're a founder
Feature complexity has the same two columns. Hard because the problem is hard is fine. Hard because the spec was vague is a cost you can delete.
If you run an AI lab or data team
Send one task and the free review will say which column its difficulty comes from, with the trajectory evidence.
Sources