How we vet who touches your tasks

The weakest point in any data programme is the moment a new person starts writing tasks for it. They know the domain, but not your rubric; they are capable, but you have not seen their work. Most programmes handle that moment with a résumé check and hope. Here is how I handle it.

1. A real trial task on your rubric

Not a generic coding quiz. A task from your programme, or one built to its spec, scored against your rubric. The question the trial answers is narrow: can this person follow your guideline exactly? Domain skill is necessary and insufficient. Plenty of strong engineers write tasks that fail review because they wrote the task they wanted to write rather than the one the spec asked for.

2. Tests decide

The trial is scored by unit tests and the rubric, with the same hidden-test checks a real task gets: does the reference solution pass in a fresh container, do the tests fail an empty submission and a hard-coded answer, is the spec followed to the letter. Opinions about whether someone "seems strong" do not enter it. This keeps the bar the same for everyone and makes it defensible when someone does not pass.

3. Only passes join

Most applicants do not pass. That is the point. A small group of people who clear the bar produces better data than a large pool with a strong average, because quality in this work is set by the weakest item in the batch, not the mean. Fewer, stronger people also means each contributor's work is known well enough to notice when it changes.

4. First batch, double-checked

Passing the trial earns a place, not trust. A new contributor's first batch is reviewed in full by a second specialist, with feedback on every item. The pass rate on that batch decides whether they continue. Someone who passed the trial but cannot hold the standard at volume leaves the programme in week one, before their work reaches you.

The public benchmark for vetting

OpenAI's GDPval is the best-documented example of expert vetting in a public evaluation. Task writers averaged 14 years of experience in their occupation, and were screened with an interview, a background check and quizzes before writing tasks that then went through about five rounds of review. That is a lab-scale version of the same principle: the filter is a process with steps, each of which removes people, and what survives is small.

What you can ask for

The names of everyone on your programme, their trial scores, and their first-batch pass rates. You can also set the trial yourself: send the task you consider a fair test and it becomes the gate. Many clients do, and it is the fastest way to align on what "good" means before volume.