For AI labs
The client changed the rubric halfway. Now what?
It happens on every programme, and it is not the problem. Unlogged changes are. A versioned change log, a same-day re-brief, a re-check of work in progress, and one written decision about delivered work keep a rubric change from becoming a dispute.
Prashant Kumar Singh · · 4 min read
In short
- Four steps the day a rubric changes: log it with a version and date, re-brief every author with examples, re-check work in progress against the new version, and agree in writing what happens to delivered work and who pays.
- Every score and every rejection carries the rubric version it was graded under; without that, nothing is comparable.
- A change nobody logged becomes a dispute nobody can settle.
The rubric will change. The client's reviewers will discover a case the rubric did not cover, or the model's behaviour will shift and a criterion that mattered stops mattering, or someone senior will read a batch and decide "good" means something slightly different. On every programme I have run, the rubric at week six is not the rubric at week one. That is healthy. The question is only whether the change is handled or absorbed.
Step 1: log it, versioned, the same day
The guideline carries a version number and a change log at the top. A change, however small, becomes a new version: v3 to v4, with the date, the reason, the exact wording before and after, and who asked for it. From then on every item submitted, every score, and every rejection records the version it was written or graded under. Without that, a rejection under v4 of an item written under v3 looks like an author error, and it is not.
Step 2: re-brief everyone, with examples
The same day, every author on the programme gets the change, with two or three real items showing what passed under v3 and would not under v4, and why. A rule stated abstractly is read differently by each author; a rule shown on examples is read the same way. If the change is large, a short recorded session replaces the message. Nobody writes a task against the new rubric until they have seen it on an example.
Step 3: re-check work in progress
Items written under v3 and not yet submitted are re-checked against v4 before they go, by their authors and then by second review. This is the step that keeps the next batch's acceptance rate from dipping, and it is cheap because the items are already open. The pre-submission checklist gets the new criterion the same afternoon.
Step 4: decide delivered work, in writing
Items already delivered and accepted under v3 are the awkward part. Three options: leave them as accepted, re-work them to v4 at the client's cost, or re-work them at mine if the change corrects something the guideline should have said all along. Which one applies is a conversation, and it ends in one email that states the decision, the number of items, and who pays. That email is what makes week twelve's invoice uncontroversial.
| Step | Same day | Artefact |
|---|---|---|
| Log | Version, date, reason, before and after wording | Change log at the top of the guideline |
| Re-brief | Every author, with real examples | Message or short recording |
| Re-check WIP | Authors, then second review | Updated pre-submission checklist |
| Delivered work | One decision: leave, rework at client's cost, or at mine | One email |
What I ask the client to do
Send changes through one person, in writing, rather than as reviewer comments on individual items. Reviewer comments are where changes are discovered, and they are a fine place to start the conversation, but a change that lives only in a comment thread has not been made. It is logged when it is in the guideline with a version number, and that is when authors start following it.
Questions people ask
- What should happen when a grading rubric changes mid-programme?
- Log the change as a new version with date and reason, re-brief every author with real examples the same day, re-check work in progress against the new version before submission, and agree in writing what happens to work already delivered and who pays for any rework.
- Why version a rubric or guideline?
- Because every score and rejection is only comparable to others graded under the same version. A rejection under a new rule of an item written under the old one is not an author error, and the version is what shows that.
- Who pays for rework after a rubric change?
- It depends on whether the change adds a new requirement (usually the client) or corrects something the guideline should always have said (usually the vendor). Either way, the decision is written down in one message with the item count.
If you're a founder
Scope changes on a product sprint follow the same four steps: written down, shown on the prototype, applied to work in progress, and quoted. The quote is the equivalent of step four.
If you run an AI lab or data team
Want the guideline template with the change log built in? Send one task for the free review and ask for it in the note.
Sources