For AI labs
Turning a research paper into a rubric tree
PaperBench graded the reproduction of 20 ICML papers with 8,316 weighted leaf criteria, each one of three types, signed off by the papers' authors. I have written and reviewed rubrics like these; 14 of 15 of my PaperBench reviews were accepted. The skill is deciding what counts and how much.
Prashant Kumar Singh · · 4 min read
In short
- PaperBench: 20 ICML 2024 papers, 8,316 leaf criteria in weighted trees, with leaves of three types (code developed, execution ran, result matched) and author sign-off on every rubric.
- A rubric tree turns "did they reproduce the paper" into thousands of pass/fail checks whose weighted average is the score, traceable to specific leaves.
- The best agents scored 21–26% at release; the rubric, not the agents, is what made that number meaningful.
"Did the agent reproduce this paper?" is not a question a grader can answer consistently. A paper has dozens of claims, several experiments, a method with many steps, and a results table with numbers that may or may not come out the same. PaperBench, published in April 2025, is the most thorough public answer to how you turn that question into something gradable, and it is the model I use when a client needs research-grade rubrics.
What PaperBench built
Twenty papers from ICML 2024, chosen to span machine learning subfields. For each one, a rubric tree: the root is "the paper is reproduced", the branches are its major contributions, and the leaves, 8,316 of them across the twenty papers, are individual checkable requirements. Every node has a weight, parents average their children, and the root score is the reproduction score. Each tree took weeks to build and was signed off by the paper's original authors.
Three types of leaf
| Leaf type | What it checks | Example |
|---|---|---|
| Code development | The implementation exists and is correct | The attention variant from Section 3.2 is implemented with the stated masking |
| Execution | The code runs and produces output | The training script runs to completion on the small config |
| Result match | The number comes out as the paper says | Accuracy on the held-out set is within 1 point of Table 2 |
The three types matter because they separate three different failures. An agent can write correct code that does not run, or run code that produces the wrong number, or get the number by a route that is not the paper's. A single "reproduced: yes/no" hides all of that. The tree shows exactly where the attempt stopped.
How the weights work
Each node's weight reflects how much of the contribution it carries, as judged by the rubric author and confirmed by the paper's authors. A core experiment weighs more than an ablation; the main result weighs more than a supplementary figure. The score at the root is the weighted average down the tree, which means a reviewer can trace any score to the leaves that produced it and argue about a specific weight rather than about the whole grade.
Grading at scale
Grading 8,316 leaves by hand per submission is not practical, so PaperBench also built SimpleJudge, an LLM-based grader, and measured it against human grading on a subset. The best configuration reached an F1 of 0.83 against human judgements. That is the same move as the previous post: hold the model judge to the human bar, measure, and use it only where it clears.
What the agents scored
At release the best agent reached a reproduction score in the low-to-mid twenties of percent; the paper reports a range of 21 to 26 depending on the configuration. The human baseline, ML PhDs given the same time, was higher. The number is less important than the fact that it is a number anyone can decompose: which leaves failed, of which type, in which part of the paper.
Building one for your own evaluation
- 1.Start from the claims. Every claim in the abstract and results is a branch.
- 2.Under each branch, write leaves of the three types until nothing is left to check.
- 3.Make each leaf binary and state the evidence that satisfies it.
- 4.Weight by importance to the contribution, argue it once in writing, then freeze it.
- 5.Get the author, or a domain expert who was not the rubric writer, to sign off.
- 6.Grade a few submissions by hand before trusting any model judge on the tree.
Questions people ask
- What is PaperBench?
- An OpenAI benchmark from April 2025 that evaluates whether AI agents can reproduce 20 ICML 2024 papers from scratch, graded with hierarchical rubrics totalling 8,316 weighted leaf criteria, each signed off by the papers' authors.
- What is a rubric tree?
- A hierarchical rubric where the root is the overall question, branches are major components, and leaves are individual binary checks with weights. Parents average their children, so the final score is traceable to specific leaves.
- How well do AI agents reproduce research papers?
- At PaperBench's release the best agent scored roughly 21–26% on the weighted reproduction rubric, below the human PhD baseline given the same time.
If you're a founder
Acceptance criteria for a complex feature benefit from the same shape: a short tree of checkable items with weights, rather than one "does it work" question.
If you run an AI lab or data team
Need rubrics for research reproduction, agentic tasks or expert evaluations? Send one existing rubric and the free review will turn its vaguest criterion into leaves.
Sources