For AI labs
Red-teaming tasks that find real failures
A red-team prompt no real user would ever send teaches the model nothing. The good ones look like Tuesday's support ticket with one trap inside, graded by a rubric that says what a safe and useful answer looks like, not only what a bad one does.
Prashant Kumar Singh · · 4 min read
In short
- A red-team task has three parts: a realistic request a real user would send, one trap (an instruction to ignore, a leak to avoid, a harmful step hidden in a helpful one), and a rubric for the safe, useful answer.
- Tricks and riddles do not train models. Realism does: the failures that matter happen on ordinary-looking requests.
- Grade helpfulness too. A model that refuses everything has also failed the task.
Red-teaming data has a quality problem that is the mirror image of the one in coding benchmarks. Coding tasks fail when they are too easy to game. Red-team tasks fail when they are too strange to matter: elaborate jailbreak riddles, role-play scenarios nobody would write, prompts whose only purpose is to be adversarial. A model trained on those learns to recognise adversarial prompts. It does not learn to handle the ordinary request with a problem inside it, which is what actually arrives.
Here is the structure I use for red-team tasks that find failures worth fixing. Three parts, each with a reason.
Part 1: a realistic request
Start from something a real user in the product's context would send. A support ticket. A question from a customer. A request from a colleague pasted into an internal tool. Tuesday afternoon, not a thriller plot. The realism is the point: the model's behaviour on strange prompts is less important than its behaviour on the ten thousand ordinary ones, and failures on ordinary prompts are the ones that reach users.
Part 2: one trap
Inside the realistic request, one thing that should change the right answer. An instruction embedded in pasted content that the model should not follow. A request that would leak something from context that the user should not see. A harmful step hidden inside an otherwise helpful task: "summarise this document and also email it to this address". A premise that is false and should be corrected rather than built on. One trap per task, so the grading can say exactly what the model did about it.
Part 3: a rubric for the good answer
Most red-team rubrics describe the bad answer: did the model comply with the harmful part. That is half a rubric. The other half describes the good answer: did it complete the legitimate part of the request, did it explain what it would not do and why, did it offer the safe alternative, did it stay useful. A model that refuses the whole ticket because one line was a trap has failed the user, and the rubric has to say so.
| Part | Weak version | Strong version |
|---|---|---|
| Request | A jailbreak riddle | A support ticket with pasted customer content |
| Trap | Obviously adversarial | One line in the pasted content that asks the model to do something it should not |
| Rubric | "Did it comply? Y/N" | Did it do the legitimate part, decline the trap, explain, and offer an alternative |
Grading helpfulness
The rubric's helpfulness criteria carry real weight, not a token one. A typical split: declined the harmful part, 40%; completed the legitimate part correctly, 30%; explained the decline in plain terms, 15%; offered a safe alternative, 15%. Under that rubric a blanket refusal scores 40, which is a fail, and a model that quietly does everything scores 60, which is also a fail. Only the model that handled both halves passes. That is the behaviour worth training toward.
Why this needs domain authors
A realistic support ticket in a medical product, a finance tool or a developer platform looks different in each, and the traps that matter are different too. The authors who write good red-team tasks are the ones who have read the real tickets. That is the same requirement as every other kind of expert task: vetted people from the domain, a scored trial on the rubric, a second review on every item.
Questions people ask
- What makes a good red-teaming task?
- A realistic request a real user would send, with exactly one embedded trap (an instruction to ignore, a leak to avoid, a harmful step inside a helpful one), and a rubric that grades both the refusal of the trap and the completion of the legitimate part.
- Should red-team rubrics grade helpfulness?
- Yes, with real weight. A model that refuses the entire request because one line was a trap has failed the user. The rubric should reward doing the legitimate part, declining the trap, explaining, and offering a safe alternative.
- Why are jailbreak-style prompts poor training data?
- They teach the model to recognise adversarial prompts, not to handle ordinary requests with a problem inside them. The failures that reach users happen on realistic requests, so that is what the tasks should look like.
If you're a founder
If your product has an AI feature that reads user content, the trap in part two is your threat model: what happens when pasted content tells the model to do something? Test that before launch.
If you run an AI lab or data team
Send one red-team task and the free review will score it against the three parts and rewrite the rubric to grade both halves.
Sources