For AI labs
Ten questions to ask any AI data vendor before you sign
Most vendor calls cover price and capacity. The questions that predict whether the data will be usable are about vetting, review, disagreement and what happens to a rejected item. Here are the ten I would want answered, with the public benchmarks that set the bar.
Prashant Kumar Singh · · 5 min read
In short
- Ask who does the work and how they were vetted. GDPval required years of experience, an interview and a background check for every task writer.
- Insist on a paid pilot graded on your own rubric before any volume commitment. DeepMind's published data practices say the same.
- The three questions vendors rarely get asked, and should: how disagreements are resolved, how AI-written submissions are caught, and what a rejected item turns into.
Buying AI training data is a procurement problem dressed as a technical one. The spreadsheet compares price per item and turnaround. The thing that decides whether the data is usable is everything the spreadsheet leaves out: who wrote it, who checked it, and what the vendor does on the day a batch comes back wrong.
These are the ten questions I would ask any vendor, including me. For each one there is a reason it matters and, where one exists, a public standard you can hold the answer against.
1. Who exactly does the work, and how were they vetted?
"Experts" is a word, not a process. Ask for the vetting steps. When OpenAI built GDPval, its task writers averaged 14 years of experience and passed an interview and a background check before writing a single task. Tasks then went through multiple rounds of review before they counted. A vendor that cannot describe its own filter in that much detail does not have one.
2. Will you run a paid pilot first, graded on my rubric?
Google DeepMind's data enrichment practices list a pilot before launch as a basic step. A pilot on your rubric, graded by you, is the only honest sample of the vendor's work. If the answer is no, or the pilot is graded by the vendor, keep looking.
3. Who reviews each item, and how many reviewers see it?
Ask whether review is a sample or every item, and whether the reviewer is a second specialist or a generalist checker. The right answer depends on the task, but "we spot-check 10%" on expert work usually means 90% of items reach you unreviewed.
4. What happens when two reviewers disagree?
They will. In GDPval, expert graders agreed on about 71% of pairwise comparisons. The vendor needs a calibration process: who adjudicates, how the rubric gets updated, and how the fix reaches the rest of the team the same day. A vendor without an answer here will send you inconsistent data and call it variance.
5. How do you catch AI-written submissions?
Contractors on major training programmes have been dismissed for using AI to do the work. Ask what the vendor does about it: tooling, spot interviews, re-derivation checks, or a plain policy with teeth. The answer tells you how much of the "expert" data was written by an expert.
6. How do you check that tests cannot be gamed?
For coding and agentic tasks, the tests are the product. Ask whether every task is run against a null agent, an empty submission and a hard-coded answer before it ships. Ask whether the grader runs outside the agent's reach. If the vendor has not heard of these checks, your tasks will be solved by shortcuts.
7. What do you report weekly?
A weekly report should carry acceptance rate, rejection causes grouped by type, rework counts and the guideline changes made because of them. If the weekly report is a count of items delivered, you are buying volume, not quality.
8. Who is my single point of contact?
One name. Someone who can answer a quality question the same day without a ticket queue. Programmes go wrong in week three, and what matters then is how fast a human with authority picks up.
9. Where does my data live?
Ask whether the vendor can work inside your tools and repositories, whether an NDA comes before any task is shared, and whether your programme will ever be named in the vendor's marketing. The answer to the last one should be never.
10. What do you do with a rejected item?
This is the question that separates vendors. A rejection is either a cost or a lesson. The good answer: it is fixed, the cause is logged, the guideline is updated, and the pre-check that would have caught it is added. The weak answer: it is redone.
Use the list as a scorecard
Put the ten questions in a column and score each vendor's answer from zero to two. A vendor that scores well on vetting and review and badly on disagreement and rejections will produce good first batches that decay. A vendor that scores well on the last four questions is set up to get better each week, which is the only kind of vendor worth a long contract.
Questions people ask
- What is the most important question to ask an AI data vendor?
- Whether they will run a paid pilot graded on your own rubric before any volume commitment. It is the only honest sample of their work, and both DeepMind's published data practices and common sense recommend it.
- How much do expert graders disagree on AI training data?
- In OpenAI's GDPval, expert graders agreed on about 71% of pairwise comparisons. That is why a vendor needs a calibration process rather than a promise that experts will agree.
- Should AI training data vendors review every item or a sample?
- For expert-level tasks, every item. Sampling 10% means 90% of what you receive was never checked by a second specialist, and errors in expert data are expensive to find downstream.
If you're a founder
The same ten questions work for any outsourced technical work: who does it, who checks it, what happens when it is wrong, and what you get each week. If your current partner cannot answer them, that is the scope problem, not the code.
If you run an AI lab or data team
Send me the vendor questionnaire you already use and I will answer it in writing, then review one of your tasks free so you can judge the work before the answers.
Sources