For AI labs
Managed team or open marketplace?
The AI data industry is moving from large pools of generalist annotators to small, vetted groups of specialists. The reason is accountability: when expert graders agree only 71% of the time, somebody has to own calibration. Here is the case for a managed team, with the evidence.
Prashant Kumar Singh · · 4 min read
In short
- The market is shifting to specialists: xAI reportedly cut 500 generalist annotators while planning to grow specialist tutors tenfold; Scale cited an industry move toward expert data work.
- Experts disagree too. GDPval's expert graders agreed on about 71% of pairwise comparisons, so calibration has to be someone's job.
- A marketplace gives you access to people. A managed team gives you one owner for vetting, review, calibration and rework.
"Why a managed team instead of a big marketplace?" is a fair question, and the honest answer starts with what the marketplaces themselves are doing.
In September 2025, xAI reportedly laid off around 500 workers from its generalist data annotation team and said it would grow its specialist tutor team tenfold. A month later Scale AI, cutting contractors, pointed to an industry shift toward higher-skill, expert data work. The large general pool is being replaced by small specialist groups across the industry, because the data frontier labs need now cannot be produced by generalists at any volume.
Experts disagree too
Specialists solve the knowledge problem and introduce a different one. In OpenAI's GDPval, expert graders from the same occupations agreed on about 71% of pairwise comparisons. Nearly a third of the time, two qualified people looked at the same pair of outputs and reached different verdicts.
That is not a flaw in the experts. It is what happens when "better" is not fully defined. The fix is calibration: a rubric that turns judgement into checkable parts, a golden answer written first, a batch where everyone grades the same items, and someone whose job is to resolve the differences and update the guideline. On a marketplace, that job belongs to nobody.
Someone has to check for AI-written work
In September 2026, 404 Media reported that contractors working on OpenAI training data had been dismissed for using AI to do the work. The incentive is obvious: per-item pay, expert-level tasks, a model that can produce a plausible answer in seconds. On a marketplace the check is a policy. On a managed team it is a person who knows each contributor's work well enough to notice when it changes.
What "managed" means in practice
| Job | Marketplace | Managed team |
|---|---|---|
| Vetting | Platform-level tests, often generic | Scored trial on your rubric before anyone joins |
| Review | Optional, usually sampled | Second specialist on every item |
| Disagreement | Majority vote or none | Calibration batch, adjudication, guideline update |
| AI-written work | Policy | A person who knows each contributor's output |
| Rework | Re-queued | Fixed, cause logged, pre-check added |
| Contact | A ticket queue | One accountable person |
When a marketplace is the right call
Fairness demands the other side. For high-volume, low-ambiguity labelling where the rubric is a lookup and the risk of a wrong item is low, a marketplace is cheaper and the lack of calibration costs little. The managed model earns its price when tasks need judgement, when the rubric is still being learned, and when a wrong item is expensive to find later. Benchmark tasks, expert evaluations and RL environments are all in the second category.
The way to find out which you need is a pilot: ten tasks, your rubric, graded by you. If the marketplace result holds, use it. If it does not, you have the comparison.
Questions people ask
- Why are AI companies moving away from generalist data annotators?
- Frontier labs need expert-level data. xAI reportedly cut around 500 generalist annotators in September 2025 while growing specialist tutors, and Scale AI cited an industry shift toward higher-skill expert data work.
- How often do expert graders agree on AI outputs?
- In OpenAI's GDPval, expert graders agreed on roughly 71% of pairwise comparisons, which is why calibration and adjudication have to be someone's explicit job.
- What does a managed AI data team do that a marketplace does not?
- Owns vetting on your rubric, second-specialist review of every item, calibration when graders disagree, checks for AI-written submissions, and rework with a logged cause. One accountable person instead of a queue.
If you're a founder
The same trade-off applies to hiring a freelancer marketplace versus one accountable builder. Access to people is not the scarce thing. Someone who owns the outcome is.
If you run an AI lab or data team
Test the claim on ten tasks. Send one task first for the free review and judge the depth of the feedback before deciding.
Sources