For AI labs
January pilot: ten tasks in week one
Start the year with a graded pilot on your rubric. A free written review of one task first, then ten tasks in one week, pre-checked before submission, with a report of acceptance, rejection causes and fixes. Priced by complexity, agreed in writing, no lock-in.
Prashant Kumar Singh · · 4 min read
In short
- Step 1: send one task for a free written review within 48 hours. Step 2: ten tasks in one week on your rubric, pre-checked before submission. Step 3: a report and your decision.
- What you get: tasks, reference solutions, hidden tests, pass/fail runs on real models, and a written report. Priced by complexity, agreed before day one.
- Scale only if the numbers hold. No retainer, no minimum beyond the ten, no lock-in.
January is when task programmes get planned and when the plans are made on the strength of rate cards. A pilot in the first two weeks of the year changes what the plan is based on: ten tasks graded by your own reviewers, with the acceptance rate and the review hours measured on your rubric. This post is the whole January pilot, so the decision to run one is easy to make.
Step 1: the free review
Send one task, rubric or test harness, under NDA or sanitised. Within 48 hours you get a written review: where an agent could shortcut it, where the spec admits two readings, which hidden-test patterns it is missing, whether the environment will drift. Nothing is owed, and the review is yours to apply to your own tasks whether or not the pilot follows.
Step 2: ten tasks in one week
Ten tasks in your domain, to your spec, on your rubric. Each one ships with what a task needs to clear review: a pinned environment by digest, a reference solution re-run ten times in a fresh container, hidden inputs generated at grading time, a null agent and a fake-success agent both scoring zero, a second valid solution passing, and a second specialist's review of the spec and tests. Each task is also run against real models, with the trajectories read and the failures grouped, so you get pass rates with reasons, not just numbers.
What you receive
| Deliverable | For each task |
|---|---|
| Task | Instruction, environment, data, pinned by digest |
| Reference solution | Verified ten for ten in a fresh container |
| Hidden tests | Final-state, property-based, grader out of reach |
| Model runs | Pass rates on real models, trajectories read, failures grouped |
| Review notes | Second specialist's checks, with what was changed |
| Report (end of week) | Acceptance, rejection causes, rework, guideline changes |
Step 3: your decision
You grade all ten against your rubric. If the numbers hold, we scale with the same gates and the same Friday report. If they are close, one more batch of ten with the guideline fixes applied. If they do not hold, you keep the ten tasks, the report and the review, and you have spent a week. The point of a pilot is that the decision is yours and the evidence is specific.
What it costs
Priced by task complexity, a repository task with a multi-file fix costs more than a single-script data task, and agreed in writing before day one. There is no retainer, no minimum beyond the ten tasks, and nothing that continues unless you choose it. The cost-per-accepted-task arithmetic from an earlier post falls straight out of the pilot's numbers.
Domains
Terminal and repository tasks, data engineering and analytics tasks, numerical and ML code, research reproduction rubrics, agentic environments, red-team tasks with realistic traps. If your domain is adjacent to these, the free review is where to find out whether the pilot would work; it will say so plainly if the fit is wrong.
Questions people ask
- What is included in a ten-task pilot?
- Ten tasks to your spec on your rubric, each with a pinned environment, a verified reference solution, hidden tests, runs on real models with trajectories read, and second-specialist review, plus an end-of-week report of acceptance, rejection causes and fixes.
- How much does an AI data pilot cost?
- It is priced by task complexity and agreed in writing before the week starts. There is no retainer, no minimum beyond the ten tasks and no lock-in afterward.
- What happens after the pilot?
- You grade the ten tasks and decide: scale with the same review gates, run one more batch with the guideline fixes, or stop and keep the tasks, the report and the review.
If you're a founder
The product equivalent is live too: a free teardown now, and a sprint that is finished before the end of January.
If you run an AI lab or data team
Send one task this week. The review is back in 48 hours and the pilot can start the Monday after.
Sources