For founders and AI labs
What I'd do for you in December
A month of playbooks in one place. For AI labs: tasks that survive review, rubrics graders agree on, environments audited for reward hacking. For founders: a one-page AI feature brief, an MVP cut to one flow, onboarding without walls. The numbers behind each, and how to start.
Prashant Kumar Singh · · 3 min read
In short
- For AI labs: tasks with pinned environments and ungameable hidden tests, rubrics built from checkable parts, and RL environments audited with null and fake agents.
- For founders: a five-section AI feature brief, an MVP cut to the one flow that proves demand, and onboarding that shows value before the wall.
- The record: 90% acceptance and 4.89/5 quality on AI training work; 22+ products shipped, 98% on the agreed date. Both paths start free.
Thirty days ago I started posting one piece a day on two subjects: building AI products, and building the data that trains and tests AI models. This is the recap, organised by what I would do for each kind of reader in December, with links to the full write-ups.
If you run an AI lab or data team
- Tasks that survive review. Pinned environments, reference solutions re-run before submission, hidden tests that fail hard-coding, empty runs and null agents. What gets a benchmark task accepted and the five checks cover the routine; 45 of 50 Terminal-Bench-style tasks were accepted with it.
- Specs nobody can misread. End state, exclusions, a worked example, checkable definitions, a ten-task pilot of the spec itself. Write task specs nobody can misread.
- Rubrics graders agree on. Split, weight, golden answer first, two graders one verdict. How to build a rubric graders agree on.
- Environments audited for reward hacking. Isolate the grader, hide answers, run null and fake agents, read transcripts. Reward hacking starts in the environment.
- A vetted, managed team. Scored trial on your rubric, second review on every item, calibration, one accountable owner. How we vet and what 90% acceptance takes.
If you are a founder or product team
- A one-page AI feature brief. User and job, data you have, three good answers, one metric, what happens when it's wrong. Scope any AI feature on one page.
- An MVP cut to one flow. Sign up, core action, result. Everything else waits for evidence. Cut your MVP to one flow.
- Onboarding without walls. Show a result first, ask one question, let people try before the email wall. Every extra form field costs you users and what a free teardown looks like.
- AI features people trust. Real sources, the user's own data, a design for the wrong answer. What makes users trust an AI feature.
- Decision-first design. One action per item, evidence one click away. From data to decisions in seconds.
The numbers
| AI training and evaluation | Product work |
|---|---|
| 90% acceptance across programmes | 22+ products shipped |
| 4.89/5 average quality rating | 98% delivered on the agreed date |
| 45 of 50 Terminal-Bench-style tasks accepted | $2,000 design sprint, $3,000 MVP sprint, fixed |
| 7 of 9 SWE-bench · 14 of 15 PaperBench reviews | Weekly written update, one person to message |
December
It is the best month of the year to start a build. Fewer meetings on both sides, faster replies, and a product or a pilot finished before January. Thank you to everyone who read, replied and sent things this month. If one of these posts described your problem, the next step is a message.
Questions people ask
- What services do you offer AI labs?
- Benchmark and training task authoring, task and rubric review, and RL environment audits, delivered by a managed, vetted team with second review on every item. It starts with a free written review of one task within 48 hours.
- What services do you offer founders?
- A free one-page product teardown, then a $2,000 design sprint or a $3,000 fixed-price MVP sprint, with weekly written updates and flex support after launch.
- What is your track record?
- 90% acceptance and a 4.89/5 average quality rating on AI training work, including 45 of 50 Terminal-Bench-style tasks accepted; 22+ products shipped with 98% delivered on the agreed date.
If you're a founder
Send the thing you are working on. The teardown will tell you which of the playbooks above applies to it first.
If you run an AI lab or data team
Send one task. The review will show which of the checks above it passes and which it does not.
Sources