For AI labs
Price per accepted task, not per task
A cheap task that gets rejected costs you twice: the fee and the reviewer-hours spent rejecting it. Terminal-Bench 2.0 accepted 39% of submissions at roughly three reviewer-hours each. The number to compare vendors on is cost per accepted task, and the arithmetic is short.
Prashant Kumar Singh · · 4 min read
In short
- Cost per accepted task = (fee × tasks submitted + review hours × hourly cost) ÷ tasks accepted.
- At 40% acceptance a $100 task costs $250 per accepted task before review time. At 90% acceptance a $150 task costs $167.
- Every rejected task also burns your reviewers' hours; Terminal-Bench 2.0 spent about three per task. The cheaper rate is usually the more expensive vendor.
"Your per-task rate is higher than the other vendor's." Probably true, and it is the wrong number to compare. A per-task rate is the price of a submission. What a lab needs is accepted tasks, and the price of those depends on the acceptance rate and on what rejections cost to find. This post does the arithmetic, with public numbers where they exist.
The public baseline
Terminal-Bench 2.0's paper reports that the benchmark accepted 89 of 229 submitted tasks, about 39%, after a review process that took roughly three reviewer-hours per task. Those are numbers from a well-run benchmark with motivated contributors. A vendor without pre-submission gates will often do worse. A rejected task still consumed its three reviewer-hours; that is the cost that never appears on the vendor's invoice.
The formula
Fee and reviewer hours are per submission because every submission costs both, accepted or not. The denominator is what you actually get.
Before review time
| Vendor | Fee per task | Acceptance | Cost per accepted task |
|---|---|---|---|
| A | $100 | 40% | $250 |
| B | $150 | 90% | $167 |
Vendor B's rate is 50% higher and its accepted tasks are a third cheaper. That is before counting anyone's review time.
With review time
Add three reviewer-hours per submission at a conservative $80 an hour for an expert reviewer. Each submission now carries $240 of review cost on top of its fee.
| Vendor | Per submission (fee + review) | Acceptance | Cost per accepted task |
|---|---|---|---|
| A | $100 + $240 = $340 | 40% | $850 |
| B | $150 + $240 = $390 | 90% | $433 |
The gap doubles. Vendor A's accepted tasks cost almost twice Vendor B's, and that is with identical review effort per submission. In practice a vendor with pre-submission gates also produces submissions that are faster to review, which widens it further.
What the arithmetic is really measuring
The acceptance rate is a proxy for whether the vendor runs the checks reviewers run, before the reviewers do. Reference solution re-run in a fresh container. Hidden tests against null and fake agents. A second valid solution passing. A second specialist reading the spec. Each one costs the vendor time and each one moves work from your reviewers' queue to the vendor's. A higher rate pays for that, and the arithmetic says it pays for itself at any acceptance gap wider than a few points.
What to ask a vendor
- What was your acceptance rate on your last three programmes, and will you show the reports?
- How many reviewer-hours per submission should we budget on your work?
- Which pre-submission checks do you run, and can we see the script?
- Will you run a ten-task pilot so we can measure both numbers on our own rubric?
The last question is the one that settles it. Ten tasks, your rubric, your reviewers' hours logged. The cost per accepted task falls out of the pilot, and it is the only number worth putting next to a rate card.
Questions people ask
- How do you compare AI data vendors on price?
- By cost per accepted task, not fee per task: (fee plus your review cost) per submission, divided by the acceptance rate. A $100 task at 40% acceptance costs $250 per accepted task; a $150 task at 90% costs $167, before review time.
- What is a typical acceptance rate for benchmark tasks?
- Terminal-Bench 2.0 accepted about 39% of submissions (89 of 229) with roughly three reviewer-hours per task. Vendors with pre-submission checks can sustain around 90%.
- Why do rejected tasks cost more than their fee?
- Because every submission consumes reviewer time whether or not it is accepted. At three expert reviewer-hours per task, a rejected submission costs the lab hundreds of dollars in review before any fee is counted.
If you're a founder
The same arithmetic applies to hiring: a cheap hourly rate with half the work redone costs more than a fixed price that ships once.
If you run an AI lab or data team
Send your numbers, fee, acceptance rate, review hours, and I will show the cost per accepted task on your programme. The free task review comes with it.
Sources