For AI labs
Run it ten times. Flaky tests ruin benchmark tasks
Google's testing team found almost 16% of its tests had some level of flakiness, and 84% of pass-to-fail transitions involved a flaky test. In production, a flaky test is annoying. In a benchmark, it grades luck. Before a task ships, the reference solution runs ten times, and it is ten for ten or it does not go.
Prashant Kumar Singh · · 4 min read
In short
- Google found about 1.5% of test runs flaky, almost 16% of tests showing some flakiness, and 84% of pass-to-fail transitions involving a flaky test.
- Flakiness hides in timeouts, ordering, random seeds, network access and the clock. Fix: seed everything, freeze time, remove network, assert on state rather than timing.
- Ten runs of the reference solution in a fresh container before submission. Ten passes or the task does not ship.
Google's testing team published the numbers in 2016 and they have not stopped being relevant. About 1.5% of all test runs at Google reported a flaky result. Almost 16% of tests showed some level of flakiness. And 84% of the transitions from pass to fail in their continuous integration involved a flaky test rather than a real regression.
That was a company with enormous engineering discipline, on production tests. A benchmark task is written by one person, run in a container, and used to compare models. If its tests are flaky, the benchmark is grading luck, and no amount of model improvement will show up cleanly in it.
Where flakiness hides
- Timeouts. A test that asserts something happens within two seconds passes on the author's machine and fails on a loaded reviewer harness.
- Ordering. Dictionary iteration, set ordering, filesystem listing, parallel completion: any order that is not guaranteed will eventually change.
- Random seeds. A test that samples without a fixed seed passes most of the time. "Most" is the problem.
- Network. Anything that reaches outside the container depends on something outside the container.
- The clock. Dates, durations, "today", daylight saving, the end of the month.
- Floating point. Sums in different orders differ in the last bits; exact equality on floats flakes.
Three fixes
Fix 1: seed everything, freeze time, remove the network
Every random source gets a fixed seed in the test. Time is frozen with a library or an injected clock. The container has no network, so anything that needs one fails immediately and loudly at authoring rather than occasionally at review. These three remove most flakiness before it starts.
Fix 2: assert on state, not timing
Check that the file exists, not that it appeared within two seconds. Check that the queue is empty, not that it emptied before the other thread finished. Check the result against a tolerance derived from the numerics, not exact float equality. A test that asserts on the final state of the world is deterministic by construction; a test that asserts on when things happened is not.
Fix 3: ten runs before submission, scripted
The last step before a task ships: a script that builds a fresh container from the pinned image, runs the reference solution, runs the tests, and repeats ten times. Ten passes, or the task goes back. The script is not optional and it is not a manual loop; it is a command that prints a table. A task that passes nine of ten is a flaky task, and the one failure is the review that would have rejected it.
| Flake source | Fix |
|---|---|
| Timeouts | Assert on state; generous or no time bound |
| Ordering | Sort before comparing; never rely on iteration order |
| Random seeds | Fixed seed in every test |
| Network | None in the container |
| Clock | Frozen time; no "today" in tests |
| Floating point | Tolerance from the numerics, in a comment |
| Everything else | Ten runs in a fresh container, scripted |
Why benchmarks care more than production does
A production flake costs a re-run. A benchmark flake costs a data point that gets compared across models and model versions for the life of the benchmark. One flaky task in a set of a hundred adds noise to every leaderboard built on it, and nobody downstream can see which task it was. That is why Terminal-Bench 2.0 verified every task across multiple runs before acceptance, and why the ten-run script is the last thing that runs before I submit anything.
Questions people ask
- How common are flaky tests?
- Google's testing team reported about 1.5% of test runs flaky, almost 16% of tests showing some flakiness, and 84% of pass-to-fail transitions involving a flaky test.
- Why are flaky tests worse in benchmarks than in production?
- A production flake costs a re-run. A benchmark flake adds noise to every comparison across models built on that benchmark, and nobody downstream can identify which task caused it.
- How do you remove flakiness from benchmark task tests?
- Fix every random seed, freeze time, remove network access from the container, assert on final state rather than timing, use numeric tolerances instead of exact float equality, and run the reference solution ten times in a fresh container before submission.
If you're a founder
Your product's test suite has the same flakes. A failing build that "just needs a re-run" is hiding a real regression 16% of the time, by Google's numbers.
If you run an AI lab or data team
Send one task and the free review will run it ten times and tell you whether it would have passed the gate.
Sources