For AI labs
Do your graders agree? Measure it
A 2025 study measured how much human graders agree with each other on answer accuracy: Cohen's kappa of about 0.80 across 1,994 items. That is the bar for any grader, human or model. Below it, you are not measuring the task. You are measuring the grader.
Prashant Kumar Singh · · 4 min read
In short
- Human graders agree with each other at about kappa 0.80 on answer accuracy (1,994 items, three graders each, in the Judge's Verdict study). That is the reference bar.
- Measure agreement on a calibration batch before anyone grades for real. Disagreement is a rubric gap, not a bad grader.
- Model judges are held to the same kappa. Some pass; many do not. Measure before you trust either.
Two reviewers score the same task differently. Which one is right? The honest answer is usually neither: the rubric was not clear enough for two careful people to reach the same verdict, and until that is fixed, every score the programme produces carries the grader's identity inside it.
The fix is to measure agreement, and the good news is that there is now a public number to measure against.
The reference number
A study published in October 2025, Judge's Verdict, set out to evaluate LLM judges and in doing so measured how well humans agree with each other. Across 1,994 items, each scored by three human graders for answer accuracy, the human-to-human agreement came out at a Cohen's kappa of about 0.80. Kappa corrects for chance agreement, so 0.80 is strong: well above what random scoring produces and about what careful people with a decent rubric achieve.
That number is the bar. A grader, human or model, whose agreement with the human consensus is well below 0.80 is adding noise. A grading team whose members agree with each other well below 0.80 has a rubric problem.
How to measure it on your programme
- 1.Pick a calibration batch. Twenty to thirty items that span the range: easy, hard, ambiguous, borderline.
- 2.Everyone grades them independently. No discussion, no shared notes, no looking at each other's scores.
- 3.Compute agreement. Cohen's kappa for two graders, Fleiss' kappa or Krippendorff's alpha for more. A spreadsheet is enough.
- 4.Read every disagreement. Each one is either a rubric gap or a misreading. Fix the criterion or re-brief the grader.
- 5.Repeat on the next batch. Agreement drifts as tasks change and as graders develop habits.
Disagreement is information
The reflex when two graders diverge is to decide who was right. The useful move is to ask why the rubric let them diverge. "Is the explanation clear" will split graders every time. "Does the explanation name the root cause and the file where it was fixed" will not. Every disagreement in the calibration batch is a sentence in the rubric that needs rewriting, and a rubric that has been through three calibration rounds is one that scales.
Model judges get the same test
The point of the Judge's Verdict study was to evaluate LLM judges, and its method is the one to copy: compute the model's agreement with the human consensus on the same items, and compare it with the human-to-human figure. A model judge at or near human kappa can grade at scale. One well below it cannot, however confident its outputs look. The study found a range: some judges reached the human tier; many did not. Which tier yours is in is an empirical question with a cheap answer.
| Agreement (kappa) | What it means | What to do |
|---|---|---|
| Above 0.80 | Graders are measuring the task | Scale; re-measure each batch |
| 0.60 – 0.80 | Rubric has gaps on borderline items | Read disagreements; fix criteria; re-run |
| Below 0.60 | Scores reflect the grader more than the work | Stop grading; rebuild the rubric; calibrate again |
Why this is cheap and skipped
A calibration batch costs each grader an hour and the programme a spreadsheet. It is skipped because programmes are in a hurry to produce volume, and because disagreement feels like a people problem rather than a measurement one. Measured once, it changes how everyone talks about scores: not "the reviewer was harsh" but "criterion four splits us, let's fix it". That shift is worth more than the hour.
Questions people ask
- What is a good inter-rater agreement for AI evaluation?
- The Judge's Verdict study measured human-to-human agreement on answer accuracy at a Cohen's kappa of about 0.80 across 1,994 items. That is a reasonable bar for both human grading teams and LLM judges.
- How do you measure grader agreement?
- Have every grader score the same calibration batch of 20–30 items independently, compute Cohen's kappa (two graders) or Fleiss' kappa or Krippendorff's alpha (more), then read every disagreement and fix the rubric criterion that caused it.
- Can an LLM judge replace human graders?
- Only if its agreement with the human consensus is at or near the human-to-human level, around kappa 0.80. The Judge's Verdict study found some model judges reached that tier and many did not, so it has to be measured per judge and per task type.
If you're a founder
User research has the same problem: two interviewers coding the same transcript differently. Agree the codes on three transcripts before coding thirty.
If you run an AI lab or data team
Send your rubric and a few graded items. The free review will flag the criteria most likely to split graders and suggest the rewrite.
Sources