For AI labs
Quality drops in week three. Here's what happens the same day
Every programme hits it. Acceptance was fine for two weeks and then a batch comes back with twice the rejections. The difference between vendors is whether the client hears about it from them, the same day, with the cause and the fix, or finds it in the numbers a week later.
Prashant Kumar Singh · · 4 min read
In short
- The same-day routine: pull every rejected item from the batch, group by cause (person, guideline, task type), apply the matching fix, tell the client before they see it in the numbers.
- Person: re-calibrate on gold tasks or remove. Guideline: fix and re-brief everyone that day. Task type: tighten the spec or the tests.
- Daily updates until acceptance recovers, then back to weekly. Every rejection becomes a line in the guideline and a check in the pre-submission script.
"What happens when quality drops mid-programme?" is the question a careful buyer asks, and the honest first half of the answer is: it will. Around week three, when the first contributors have settled into habits, the task mix has shifted, and the guideline has not caught up, a batch comes back with more rejections than the two before it. Every programme I have run has had a version of this week. What distinguishes programmes is the second half of the answer.
Hour one: pull every rejection
Not a sample, every rejected item from the batch, with the reviewer's note. Laid out in one table: item, author, task type, rejection reason. The pattern is usually visible within an hour, and it is almost always one of three.
Cause 1: a person
One author accounts for most of the rejections. They passed the trial, their first batch was fine, and something has drifted: speed over care, a misreading that became a habit, a new task type they are weaker on. The fix is specific to the person: re-calibrate them on gold tasks with known answers, coach one to one on the exact rejections, and if the next small batch does not recover, take them off the programme. The rest of the team is not re-briefed for one person's drift.
Cause 2: the guideline
Rejections are spread across authors and share a reason: the reviewer is applying a rule the guideline never stated, or stated ambiguously. This is the most common cause and the most fixable. The guideline is corrected that afternoon, every author is re-briefed with the exact rejections as examples, and work in progress is re-checked against the new line before submission. The client is asked to confirm the rule if the guideline's reading was defensible.
Cause 3: a task type
A new kind of task entered the mix and the spec or the tests for it are weaker than for the rest. The rejections cluster by type, not by author. The fix is upstream: tighten that type's spec, strengthen its hidden tests, add a worked example, and run the next batch of that type small until it clears.
| Pattern in the rejections | Cause | Same-day fix |
|---|---|---|
| One author, many reasons | Person | Gold-task re-calibration, coaching, or removal |
| Many authors, one reason | Guideline | Fix, re-brief with examples, re-check WIP |
| One task type, many authors | Task type | Tighten spec and tests, small batch until clear |
Same day: tell the client
Before the client sees it in their own numbers, they get a message: what dropped, by how much, the cause, what changed, and when the next batch will show whether it worked. Then daily updates until the acceptance rate is back where it was, and then the normal Friday rhythm. A quality drop reported by the vendor with a fix is a normal week. The same drop discovered by the client is a vendor review.
What the client sees in the report
The weekly report's rejection section for that week shows the dip, the cause grouping, the fix applied and the recovery curve on the following batches. It is the most useful report a programme produces, because it shows the system working rather than the numbers being good. Clients who have seen one of these tend to stay, because the question they were really asking was not "will quality drop" but "what will you do when it does".
Questions people ask
- Why does data quality drop a few weeks into a programme?
- Usually one of three causes: an individual author has drifted into bad habits, the guideline does not state a rule reviewers are applying, or a new task type has entered the mix with a weaker spec or tests.
- What should a vendor do when quality drops?
- The same day: pull every rejected item, group by author, reason and task type, apply the fix that matches the cause, and tell the client before they see it in their numbers, then update daily until the acceptance rate recovers.
- How do you stop the same quality problem recurring?
- Every rejection becomes a line in the guideline and a check in the pre-submission script, so the specific mistake is caught before submission on all future batches.
If you're a founder
A product sprint has the same moment: the week something slips. The weekly update exists so you hear it from me with a plan, not from the deadline.
If you run an AI lab or data team
Want to see the report format before committing? The free task review comes with a sample of the weekly report attached.
Sources