For AI labs
Seven ways a task environment drifts
A task that passed in October fails in December and nobody changed the task. Everything around it changed. Seven things that move between submission and review, and the pin for each one. Re-run the reference solution the day you submit.
Prashant Kumar Singh · · 4 min read
In short
- Seven sources of drift: image tag, package versions, data URLs, clock and locale, deprecated APIs, resource limits, and tests tightened after the solution was written.
- Pin the image digest, lock every package, copy data into the image, set TZ and LANG, fix memory and timeouts, avoid external APIs, and re-run the reference solution the day you submit.
- Terminal-Bench 2.0 verified each task across multiple runs in pinned environments for exactly this reason: a task that cannot be reproduced measures nothing.
"It passed on my machine" has a benchmark cousin: "it passed in October". The task is unchanged, the reference solution is unchanged, the tests are unchanged, and the task fails at review. Something in the environment moved. Here are the seven things that move, in rough order of how often I see them, with the pin that stops each.
1. The base image tag moved
python:3.12 on Tuesday is not python:3.12 on Friday. Tags are pointers; the image behind them is rebuilt with new system packages, a new patch release, a new default. Pin: reference the image by digest (@sha256:…), never by tag, and record the digest in the task's README.
2. A package minor version bumped
A pip install pandas that resolved to 2.2 in October resolves to 2.3 in December, and a deprecation became an error. Pin: a lock file with every package and every transitive dependency at an exact version, installed with --no-deps from the lock. Not >=. Not ~=. Exact.
3. A dataset URL changed
The task downloads a CSV at build time; the host moved it, rate-limited it, or updated it with one more column. Pin: copy data into the image at authoring time. No downloads at build or run time. If the data is too large, store it with the task and mount it, with its checksum recorded.
4. The clock and the locale
A test that computes "days until the end of the quarter" passes until the quarter changes. A sort that depends on locale collation passes until the image's default locale changes. Pin: set TZ and LANG explicitly in the image, and freeze time in tests that depend on it.
5. A model or external API deprecated
A task that calls an external service inherits that service's release schedule. Pin: avoid external APIs inside the task entirely. If the task is about calling one, mock it with recorded responses stored in the image.
6. Disk, memory and time limits
The reference solution used 3.5 GB on the author's machine; the review harness allows 2 GB. A test that takes 40 seconds locally hits a 30-second timeout in CI. Pin: state the limits in the task, run the reference solution under exactly those limits, and leave headroom.
7. A test tightened after the solution was written
The author improves the tests on day three and does not re-run the reference solution from day one. Pin: the reference solution runs against the final tests, in a fresh container, as the last step before submission. Every time the tests change, this step runs again.
| Drift | Pin |
|---|---|
| Image tag | Digest, recorded in README |
| Package versions | Exact lock file, --no-deps |
| Data URL | Data copied into the image, checksum recorded |
| Clock and locale | TZ, LANG set; time frozen in tests |
| External APIs | None, or recorded mocks |
| Limits | Stated in the task; reference run under them |
| Tests tightened | Reference re-run on final tests in a fresh container |
Why the benchmarks insist on this
Terminal-Bench 2.0's paper describes verifying every task across multiple runs in its pinned environment before acceptance, because a task that cannot be reproduced cannot be used to compare models. Drift is not a nuisance; it is a correctness bug in the measurement. The pins above take an hour per task to put in place. The review they prevent takes three.
Questions people ask
- Why does a benchmark task that passed earlier fail later?
- Environment drift: the base image tag was rebuilt, a package resolved to a newer version, a dataset URL changed, the clock or locale moved, an external API changed, resource limits differ at review, or the tests were tightened after the reference solution was last run.
- How do you pin a Docker environment for a benchmark task?
- Reference the base image by digest rather than tag, install packages from an exact lock file, copy data into the image, set TZ and LANG, state and test under resource limits, and avoid external APIs or mock them with recorded responses.
- When should you re-run a task's reference solution?
- In a fresh container from the pinned image, on the day of submission, and again every time the tests change. Ten runs, ten passes.
If you're a founder
Your product has drift too: the API you integrated, the model version you call, the library you pinned loosely. A lock file and a staging re-run before each release are the same discipline.
If you run an AI lab or data team
Send one task with its Dockerfile and the free review will check it against all seven and tell you which will drift first.
Sources