For AI labs
How to build a rubric graders agree on
Expert graders in OpenAI's GDPval agreed on 71% of pairwise comparisons. A rubric gives them something checkable to agree on. The method behind 14 of 15 accepted PaperBench reviews: split quality into parts, weight them, write the golden answer first, and require two graders to reach one verdict.
Prashant Kumar Singh · · 4 min read
In short
- "Is this answer good?" cannot be graded consistently; "does it produce the right total?" can. A rubric turns the first question into many of the second.
- Four steps: split quality into small checkable parts, weight them, have an expert write the golden answer before grading starts, and require two graders to reach the same verdict independently.
- Anthropic's test of a well-designed task is that experts agree independently; disagreement is a signal the rubric has a gap, not that a grader is wrong.
In OpenAI's GDPval, expert graders from the same occupation compared pairs of outputs and agreed about 71% of the time. These were people with years of experience judging work in their own field. If they diverge nearly a third of the time on "which is better", a programme that asks graders that question directly will get inconsistent data no matter who it hires.
The fix is to stop asking that question. A rubric replaces "is this good" with a list of things that are either present or not. Here is how I build one. It is the method behind 14 of my 15 accepted PaperBench reviews, where the rubric is the entire deliverable.
1. Split "good" into checkable parts
Take the vague question and keep asking "what would I look at to decide" until every answer is something a second person could verify. For a coding task: the code runs; the tests pass; the output matches the reference within tolerance; no file outside the allowed directory was changed; the explanation names the root cause. Each part is a yes or a no. If a part cannot be made binary, split it again.
2. Weight each part
Not every part matters equally. A typical split for a coding task: correct result 40%, works end to end 30%, code quality 20%, explanation 10%. The weights are a product decision and should be argued about once, in writing, before grading starts. PaperBench does this with a tree: every leaf has a weight, and parent nodes average their children, so the final score is traceable to specific leaves.
3. Write the golden answer first
Before anyone grades, an expert writes the ideal response. This does three things. It proves the task can be done. It shows what full marks look like, so graders are matching against a thing rather than an idea. And it exposes rubric parts that the ideal answer does not satisfy, which means the rubric is wrong, not the answer.
4. Two graders, one verdict
Anthropic's guide to agent evals gives the test for a well-designed task: experts should agree on the grade independently. So calibration runs before volume: two graders score the same twenty items without conferring, and every disagreement is examined. Each one is either a rubric gap (fix the criterion) or a grader misreading (fix the briefing). The rubric that comes out of calibration is the one that scales.
| Vague | Checkable |
|---|---|
| The code is clean | No function over 50 lines; linter passes with project config |
| The answer is well sourced | Every factual claim cites a document in the provided set |
| The fix is correct | Hidden tests pass; reference output matched within stated tolerance |
| The explanation is clear | Names the root cause and the file where it was fixed |
What a rubric is not
It is not a checklist of everything that could be wrong. A rubric with forty items is as hard to apply consistently as no rubric at all. Keep it to the parts that decide the grade, weight them, and move the rest into a guideline that explains the spirit. And it is not fixed: every calibration batch and every disputed verdict should change it, with a version number so everyone knows which rubric a score came from.
Questions people ask
- How much do expert graders agree when evaluating AI outputs?
- About 71% of pairwise comparisons in OpenAI's GDPval. That level of disagreement among experts is why rubrics and calibration are necessary rather than optional.
- How do you design a grading rubric for AI evaluation?
- Split quality into small binary checks, weight each one, have an expert write the golden answer before grading begins, and calibrate by having two graders score the same items independently and fixing the rubric wherever they disagree.
- What is a golden answer in evaluation?
- The ideal response to a task, written by an expert before grading starts. It proves the task is solvable, shows graders what full marks look like, and exposes rubric criteria the ideal answer does not meet.
If you're a founder
Acceptance criteria for a feature work the same way. "Users should find it easy" cannot be checked; "a new user completes the flow in under two minutes without help" can.
If you run an AI lab or data team
Send your rubric and the free review will mark each criterion checkable or not, suggest weights, and flag where two graders would diverge.
Sources