I have a side project that reads a live feed of EVE Online killmails, writes each one to Postgres, and keeps a cursor so it can pick up where it left off. It is about as unglamorous as a service gets. Read, save, advance, repeat.
Last week its tests started failing, and only on the machine that was busy. My laptop stayed green. Twelve copies of the suite running at once, each pinned to a single core, all pointed at one database, and eight of the twelve went red.
Stored two rows, wanted three. Cursor sitting at 102, wanted 103.
I knew what to do about that, because I have done it perhaps a hundred times in twenty years. Find the number, make the number bigger, get on with your day. It was 400 milliseconds. It could be 800. Nobody would ever have asked me about it.
What that test was actually claiming
The test ran the consumer under a deadline and then asserted how much work it had finished. Written out honestly, that is two separate claims wearing one assertion.
The first is about my code: given this feed, the consumer stores three rows and leaves the cursor at 103.
The second is about the afternoon. It does all that inside 400 milliseconds, on whatever hardware happens to be free, however many other processes are fighting it for the disk.
Only one of those is a statement about software I wrote. The other is a statement about a machine I do not control and never measured.
When the assertion goes red, it cannot tell you which of the two broke. That is the whole disease. A result you cannot interpret is a result you learn to ignore, and a test you have learned to ignore is worse than no test at all, because it costs the same to run and buys you nothing.
Look again at what came back. Stored two, wanted three. That is not the state of a broken consumer. A broken consumer stores the wrong thing, or nothing, or the same thing twice.
Storing two of three rows and then stopping is the state of a consumer that was interrupted, and interrupting things is precisely what a deadline does for a living. My test was describing its own timer, and I nearly filed it as a defect in my code.
Waiting for the thing to say it is done
The fix is a different question, not a bigger number. Instead of running the consumer for a length of time and then asking what it managed, run it until its own durable output says it has finished.
The consumer already publishes that. The cursor is a row in Postgres recording which sequence it has passed. So the tests now run it until the cursor reaches the value the scenario expects, and then stop it.
There is still a thirty second bound in there, but it is a failure ceiling rather than a schedule. A healthy run never goes near it. If a run ever does hit it, that is not a flake to be smoothed over, that is a hang, and I want the red.
Twelve concurrent single-core runs against one database: eight failures before, none after. Then forty concurrent runs, also clean. That is the point where I would normally have closed the issue and felt good about my morning.
The honest test stops it at the worst possible moment
I did not see the next part coming, and it is why I am writing this down instead of filing it under test tidying. A deadline-based test cancels late. It fires four hundred milliseconds in, long after the three rows have landed, while the consumer sits there with nothing left to do. That is the safest possible moment to interrupt anything.
Without meaning to, the old test had been carefully avoiding every interesting part of the shutdown path for as long as it had existed.
A condition-based test cancels the instant the work completes. Not a moment later. The condition goes true because a write landed, and the cancel goes out while the process is still inside the machinery of that write.
So the new tests do not merely remove a race. They aim at the exact window where a service is most fragile, and they hit it on every single run. Two real bugs fell out of that window inside an hour.
A clean stop that logged a crash
The command that runs this thing returns whatever the consumer returns. Straightforward, and fine, right up until a shutdown lands inside the cursor write.
The write gets cancelled, the cancellation surfaces as an error, the consumer hands that error upward, and the process exits non-zero after logging a storage failure.
Nothing was actually wrong. That is a normal stop. The cursor stays where it was, and the next start re-reads a handful of sequences the store already deduplicates.
But the exit code says otherwise, the log line says otherwise, and systemd reads exit codes. It is generous about signals: a process killed by SIGTERM counts as a clean exit, and so do SIGHUP, SIGINT and SIGPIPE.
It is not generous about a process that catches the signal, tidies up politely, and then exits one. That is an unclean exit code, and it gets recorded as a failed unit.
So every restart with unlucky timing would have painted a crash into the journal and a database error into my logs, describing a problem I did not have. I would have gone looking for it. Probably for an afternoon, some week when I had other things on.
The repair is to say what I actually mean. An error that arrives while I am the one asking it to stop is a stop, not a failure, and the check belongs on the context rather than on the error text.
A commit interrupted by a cancellation comes back wrapped several layers deep as a timeout, and matching on that string would be a guess about somebody else's error formatting, which changes whenever they feel like changing it. Whether I cancelled is not a guess. I am the one who did it.
The row was there and the counter was not
The second bug is stranger, and it is the one worth carrying out of here even if you never touch Go or Postgres. Cancelling a request while a commit is in flight does not reliably un-commit anything.
The write can be durable on the server while the client gets an error back. Postgres documents this in general terms, saying that dispatching a cancellation is no guarantee the request will have any effect.
The maintainer of the driver I use has confirmed the specific case on a public issue. The commit went out, the context expired before the response came home, and the only way to know what happened is to go and ask the database.
The driver's own source calls the cancel request inherently racy, which I found reassuring in the way that only an honest comment can be.
Which means a test that waits for a row to appear and then asserts the consumer's count of rows is comparing two numbers that were never guaranteed to agree. The row landed. The commit returned cancelled. The increment that would have counted it never ran. The consumer honestly reports one fewer than the database honestly holds, and both of them are telling the truth.
So the tests wait on the cursor instead, and not out of tidiness. The cursor is written after the counter is incremented, so anything it has passed is already counted.
That makes it the marker that actually implies the state I want to assert. A row's existence implies nothing at all about the counter.
Three tests had to move onto it. The third only failed at twenty four way contention, which is not a workload anybody runs on purpose. It is a workload you create by running your own suite twenty four times at once to see what falls out of it.
The retry button is a redaction
I want to be fair to the other side here, because the other side is not stupid and it has far more data than I do. At real scale, most flaky tests genuinely are the test.
Google has published its own numbers. Roughly one in six of its millions of tests carries some level of flakiness, and the large majority of transitions from passing to failing come from that flaky minority rather than from anybody breaking anything.
Blocking a very large engineering organisation on somebody's timing assumption is expensive in a way that is easy to put a figure on, and rerunning the case costs almost nothing.
That is why the CI vendors sell automatic retries, and where they explain themselves the reason is throughput: recover from an intermittent failure without paying for the whole suite again. At their scale I would probably make the same trade.
But watch what a retry does to the failure I actually had. It reruns the case on a machine that is, by definition, now slightly less busy. It gets a green. It files the incident under noise.
The two bugs underneath survive completely untouched, and they survive precisely because they only appear under contention, which is the one condition a rerun is designed to wash out.
The class of defect a retry deletes is the class that turns up in production on a bad day, when the disk is slow and something else is restarting.
The research is less comfortable for the retry habit than the retry habit assumes. One empirical study went through two hundred and one real flaky-test fixes across fifty one Apache projects.
Roughly a quarter of them were fixed by changing the code under test rather than the test, and almost all of those turned out to be genuine bugs in the system. Among the failures caused by concurrency, nearly a third were non-determinism in the software itself.
The same study looked at how people fix these things. Adding a sleep never fully removed the flakiness, it only lowered the rate. Waiting on an actual condition removed it outright in more than half the cases.
Martin Fowler got to the same place from first principles fifteen years ago and put it more bluntly than I would dare: non-deterministic tests are "useless" and "a virulent infection".
There is also a version of this argument that depends entirely on how big you are. A large organisation is choosing between a flake budget and a thousand blocked engineers. I am one person, I carry my own pager, and the only person I block is me. Reading the red is the cheapest thing I do all week.
A clock is not an assertion
The number in a wall-clock wait is a guess about hardware you do not control, dressed up as a check on code you do.
It gets written while you are trying to finish something else, and every time it fails you are invited to make it bigger, which always works, which is exactly what makes it so hard to stop doing.
The Go team put this well while arguing for a better way to test timing. You can make a sleeping test slower to make it less flaky, or faster to make it more flaky, but you cannot make it both fast and reliable.
I got two production bugs out of this. Neither was ever going to be found by a person reading the code, because the code looks correct and mostly is.
They were found by a busy machine interrupting a write at exactly the wrong instant, over and over, until the pattern stopped being deniable. Contention is a fuzzer I already own and had been carefully switching off.
So the next time something goes red only under load, before you touch the number, read the failing state and ask what it is really describing. Mine was describing a service that got interrupted mid-write and then reported itself as a crash.
It had been trying to tell me that for a while. I had been telling it to take longer.
