For AI labs
What a 90% acceptance rate actually takes
Four gates, every item, every batch. A scored trial on the client's rubric, a second specialist on every item, calibration when grading drifts, and sign-off that turns each rejection into a guideline fix. Not talent. A system.
Prashant Kumar Singh · · 4 min read
In short
- Gate 1: a scored trial on the client's rubric. Gate 2: second-specialist review of every item. Gate 3: calibration when grading drifts. Gate 4: sign-off that converts each rejection into a guideline change.
- Expert graders in GDPval agreed about 71% of the time; the calibration gate exists because disagreement is normal, not exceptional.
- The result across programmes is 90% acceptance and a 4.89/5 average quality rating, driven by the gates rather than by individual brilliance.
Across the AI training and evaluation programmes I have run, 90% of submitted work has been accepted, with an average quality rating of 4.89 out of 5. People sometimes read that as a claim about the contributors. It is a claim about the routine. Here it is, gate by gate.
Gate 1: a scored trial on the client's rubric
Nobody joins a programme on a résumé. They complete a task from the programme, scored by its tests and rubric, and only passes join. The previous post covers this in detail. Its contribution to the acceptance rate is simple: the people who would produce the rejected items mostly never start.
Gate 2: a second specialist reviews every item
Not a sample. Every item is reviewed by a second person with the domain skill to find real errors, before it is submitted. The review follows the same checklist the client's reviewers use: reference solution re-run in a fresh container, hidden tests run against an empty and a hard-coded submission, spec compared line by line to the task. Most items that would have been rejected are caught here, and fixed before the client ever sees them.
Gate 3: calibration when grading drifts
Even expert graders disagree. In OpenAI's GDPval, experts agreed on about 71% of pairwise comparisons. So on every programme a calibration batch runs when the data suggests drift: several reviewers grade the same items, the disagreements are adjudicated, and the rubric or guideline is updated and re-briefed the same day. Without this gate, two careful reviewers slowly diverge, and the client's reviewers see inconsistency they cannot trace.
Gate 4: sign-off, and the rejection loop
Final sign-off is where the items that still get rejected by the client are turned into something useful. Each rejection is logged with its cause, the guideline gains a line that would have prevented it, and the pre-submission checklist gains a check. The same mistake does not come back in week six. This is the gate that makes the acceptance rate rise over a programme rather than decay.
| Gate | Catches | Cost if skipped |
|---|---|---|
| Scored trial | People who cannot follow the rubric | Rejections from the first batch onward |
| Second review | Spec drift, weak tests, broken environments | The client's reviewers become your QA |
| Calibration | Graders diverging | Inconsistent batches, disputed verdicts |
| Sign-off loop | Repeat mistakes | The same rejection every week |
What the system costs
Second review on every item is the expensive gate, and it is the reason a per-task rate from a managed team is higher than a marketplace's. The arithmetic is covered in a later post: at a 40% acceptance rate, a cheap task costs more than twice its price per accepted task, before counting the client's reviewer hours spent rejecting it. The gates are what you are paying for, and the acceptance rate is what they return.
Questions people ask
- What is a good acceptance rate for AI training tasks?
- Public benchmarks give a baseline: Terminal-Bench 2.0 accepted about 39% of submitted tasks. A managed process with a second-specialist review on every item can sustain around 90%, which is the rate across my programmes.
- What are the four quality gates for AI training data?
- A scored trial on the client's rubric before anyone joins; second-specialist review of every item; a calibration batch when grading drifts; and sign-off that turns each rejection into a guideline and checklist change.
- Why do you need calibration if your reviewers are experts?
- Because experts disagree. In OpenAI's GDPval, expert graders agreed on about 71% of pairwise comparisons. Calibration catches the drift before the client's reviewers do.
If you're a founder
The product equivalent of a quality gate is a written update every week and a working build every few days, so problems are found in days rather than at handover.
If you run an AI lab or data team
Want the gates on your programme? Start with the free review of one task; it applies Gate 2 to your work so you can see what it catches.
Sources