For AI labs
Try us on ten tasks before you commit
Google DeepMind's published data practices say to run a pilot before launching a data programme. So do we. Ten tasks on your rubric in one week, graded by you, with a written report. You scale only if the numbers hold.
Prashant Kumar Singh · · 3 min read
In short
- DeepMind's data enrichment best practices recommend piloting a task with a small batch before launching at volume.
- A pilot is ten tasks on your rubric in one week, graded by you, with a report of acceptance, rework and the guideline changes made.
- The pilot is priced by complexity and ends with a decision in your hands: scale, adjust or stop.
Nobody hires an engineer without a trial task. Yet data programmes are regularly started at volume, on the strength of a deck and a rate card, and the first real sample of the vendor's work arrives as batch one with a thousand items in it.
Google DeepMind's best practices for data enrichment include piloting a task before launch: run a small batch, check that the instructions produce the data you expected, fix what did not work, then scale. That is the model I run, and this post is exactly what the pilot looks like so you can judge it before booking one.
Day 1: free review of one task
Before the pilot, send one task, rubric or test suite. Under NDA, or a sanitised sample if you prefer. Within 48 hours you get a written review: where an agent could shortcut it, where the spec is ambiguous, what the tests miss. This costs nothing and it tells you what level of review the pilot will apply to your work.
Week 1: ten tasks on your rubric
Ten tasks in your domain, written to your spec and graded against your rubric. Each task carries what a task needs to pass review: a pinned environment, a reference solution re-run before submission, hidden tests that fail hard-coding and empty runs, and a second-specialist review before it reaches you. You grade every one. Not a sample, every one, because the point is to see the floor as well as the ceiling.
End of week 1: the report
A written report with acceptance rate, every rejection grouped by cause, rework count, and the changes made to the guideline because of them. This is the same report you would get weekly at volume, so you are also judging the reporting, not only the tasks.
Then: your decision
| Outcome | What happens |
|---|---|
| Quality holds | Scale to volume with the same review gates, priced by complexity, weekly reports |
| Quality is close | One more batch of ten with the guideline fixes applied, before any volume |
| Quality does not hold | You have lost a week, not a quarter, and you keep the ten tasks and the report |
Why I offer it this way
A pilot is as useful to me as to you. Ten tasks tell me whether your rubric is clear enough to scale, whether your domain needs a specialist I have not yet lined up, and whether the two of us agree on what "accepted" means. Programmes that skip the pilot find all of that out in week three, which is where most of the industry's quality problems live.
You can stop after any step. That is the whole point.
Questions people ask
- What is a pilot in AI training data procurement?
- A small batch of tasks, typically ten, written to the client's spec and graded on the client's rubric before any volume commitment. DeepMind's published data practices recommend it before launching a data programme.
- How long does an AI data pilot take?
- One week for ten tasks, after a free 48-hour review of one task. The pilot ends with a written report and a scale, adjust or stop decision.
- Who grades the pilot tasks?
- You do. Every task, against your own rubric. A pilot graded by the vendor is not a pilot.
If you're a founder
The equivalent for product work is the free teardown followed by a fixed-price design sprint. Small, graded by you, before anything larger.
If you run an AI lab or data team
Start with the free review. Send one task and judge the review before deciding whether the pilot is worth a week.
Sources