A deploy that passed on retry is not a fluke
A failed build that succeeds on the second try feels like luck. Here is how to tell a one-off from a slow leak before it takes you down.
One of our deploys failed with a "no space left" error. We retried, and it passed. That is the trap: the retry worked, so it looked like a fluke.
A retry that passes proves very little
Build systems keep caches so they stay fast, and caches grow. When a machine runs short of space, the first failure often lands at the very last step, where there is just enough room left for one more run. A retry can squeak through. Nothing was fixed; the slack was simply used up.
Watch where it fails, not whether it fails
The signal is the position of the failure. If the same job later fails earlier than before, the pressure has grown.
A failure that moves earlier is not a second fluke. It is the same problem advancing, and another retry will not hold.
What we do now
- Never retry in a loop on a space error. First compare where it failed with last time.
- If it failed earlier, run one clean build that throws the old caches away.
- Expect that build to be noticeably slower. The slowness is how you know it really started from scratch.
- Turn caching back on straight afterwards.
What it means for you
Treat a failure that passes on retry as a question, not an answer. Where it failed matters more than whether it failed, and a run of retries that keep working is the problem growing quietly. If you want a second pair of eyes on your delivery setup, get in touch.