The Vulnerability Was the Easy Part

The Vulnerability Was the Easy Part

The patch took ninety seconds.

A docker compose pull, a docker compose up -d, and the reverse proxy sitting in front of twenty containers went from a version with a published response-poisoning bug to one without it.

Two seconds of downtime. I timed it, because I was nervous.

Everything else that went wrong today was something I added afterwards to make sure I would never have to do that again.

The advisory that landed this morning

Traefik published it today. Cross-user response poisoning through a proxied CONNECT request, scored 7.0, no CVE assigned to it.

The mechanism is the kind I find genuinely unpleasant. Traefik keeps a pool of keep-alive connections to your backends and reuses them across requests, which is the entire point of a connection pool.

Send it an HTTP/2 or HTTP/3 CONNECT that it forwards to an HTTP/1.1 upstream. Have that upstream answer with a non-2xx, keep the connection alive, and never drain the request body.

The socket goes back into the shared pool out of step with itself.

The next client to borrow that socket reads a response meant for somebody else. Possibly an authenticated one.

Nothing throws. Nothing logs an error. The proxy believes it did its job, the backend believes it did its job, and one of your users quietly reads another user's page.

Versions 3.6.0 through 3.6.23 are affected, fixed in 3.6.24, with 2.11.53 and 3.7.9 covering the older and newer lines.

I have written publicly about this exact setup. One box, Docker, Traefik in front, everything else behind it. That is my stack, and I have recommended it to strangers.

When the thing you recommended turns out to have a hole in it, you go and look at your own box first.

Four weeks of uptime is not four weeks of health

Mine was running 3.6.21, built on the 10th of June. Up four weeks.

Here is the part that stung. My compose file pins traefik:v3.6, a floating tag. Not a digest, not a patch version.

Which means the fixed build was one docker compose pull away from me for as long as it had existed, and any restart at all would have picked it up.

There was no restart. Nothing on that box pulls anything on a schedule. The proxy had been running continuously and correctly for four weeks, and I had been reading that number as a good sign.

Uptime is not health. It is the absence of change, and the absence of change is exactly what leaves you sitting on an old build. The number I was quietly proud of was the number describing the problem.

Ninety seconds

Back up acme.json, pull, up -d, watch it come back. Two seconds to the first successful request. Twenty containers behind it, all fine, no errors in the log.

3.6.21 to 3.6.24. That was the whole fix.

If this stopped here it would be a boring and useful reminder to go and check your proxy. It does not stop here, because I then decided to solve the actual problem.

Then I automated it, and that is where the incidents started

The real failure was never the version number. It was that nothing would have told me.

I found out because a scan I run happened to surface the advisory on the morning it was published. That is luck wearing the costume of a process.

So: something that watches the proxy image and updates it when there is something to update. Small, contained, one container. Watchtower is the obvious answer and I have used it before.

Every single problem for the next hour came out of that decision. Not one of them came from the vulnerability.

The tool I reached for had been dead for seven months

Watchtower was archived on the 17th of December 2025. Read-only. The final release landed about a month before that.

Nothing tells you this. The image still pulls. docker run still works exactly as it always did.

Every tutorial and every forum answer still tells you to use it, because tutorials do not expire.

The repository carries a banner, which you see only if you go and look at the repository, and nobody goes and looks at the repository for a tool they already know how to use.

There is a maintained fork and switching to it is a one-word change to the image name. Fine.

But I would have deployed the dead one without blinking. I only checked because I was about to give something root-equivalent access to the Docker socket on a production box, and that felt like it deserved five minutes.

That is the honest reason I caught it. Not diligence. Nerves.

An updater with no scope is a thing that restarts your production

I am not typing most of this myself these days. I work through an agent, and have done for the better part of a year, and the division of labour is that it does the mechanics while I decide what the mechanics should be.

While working out the cron format it ran the updater once with the scoping flag missing.

Watchtower with no scope does not watch one container. It watches every container it can see.

It scanned all twenty. It updated and restarted one site's entire stack: the application, its database, its backup sidecar. A live site, in the middle of the afternoon.

The only reason it stopped at one was that my private registry images returned 401 and it could not pull them. An authentication failure as an accidental blast shield.

Everything came back healthy, the site returns 200, the logs are clean. It cost that site a few seconds of downtime it had no reason to spend. Good outcome, entirely unearned.

I want to be careful with the lesson, because "the agent did it" is the shallow version and it is not what happened. I would have made the same class of mistake.

Anyone who has ever run a one-off command to see what a flag does has run it against more than they meant to.

The real thing is this. An updater holding the Docker socket is production infrastructure from the moment it first runs, and I was treating it as something I was still configuring.

There is no configuring phase. It had the socket on the first invocation.

scanned=1

After that I stopped assuming and started proving.

The scoped version restricts itself by container label, and the failure mode of a label filter is silent in both directions. Get the label name wrong and it either watches nothing or watches everything.

The startup log says the same cheerful thing either way. It tells you the schedule. It does not tell you what it can see.

So I ran it in monitor-only mode and read the number it reports back.

scanned=1

One container. The proxy. Not twenty.

That is the difference between a configuration I believe in and a configuration I have checked, and it is one line of output.

I also ran it once with a deliberately invented notification scheme, to make sure the check was capable of failing. It died with unknown service, which is what I wanted to see.

A test that cannot fail is not evidence of anything. I would rather discover my test is inert on a Monday afternoon than during the next advisory.

Initialised is not delivered

Last piece. Notifications, so that a 4am update is not silent.

I wired the webhook through an environment file so the secret stays out of the compose file, started it, and the log said Using notifications: slack.

That line means the URL parsed. It means a string matched a scheme the binary recognises. It does not mean Slack has ever accepted anything, that the channel exists, or that a human will ever see a message.

I nearly stopped there. The line looks exactly like success.

What happened instead was worse and considerably more useful. The webhook had been pasted in its raw form, and the tool wants its own URL scheme, so the container went into a crash loop.

Fatal on startup, restart, fatal again, every few seconds.

I had already checked that an empty value degrades quietly to no notifications. A malformed one does not degrade at all.

And every one of those fatal log lines printed the full webhook URL. A crash loop turns out to be a very efficient way of copying a credential into every log you own. That webhook is being rotated.

The proxy never wavered through any of it, which is the one piece of design that held: the thing that updates Traefik cannot take Traefik down.

The startup message eventually arrived in the channel. That is the point at which I believed it. Not when it initialised.

The thing you build to stop the outage

Nothing that broke today was the vulnerability. The vulnerability was ninety seconds and a version number.

What broke was an archived dependency, an unscoped updater restarting a live site, a crash loop, and a credential in a log file.

Four incidents, all of them mine, all of them introduced by the machinery for making sure the first problem never happens again.

I am not arguing against automating it. The alternative is precisely what I already had, which was a proxy quietly running a vulnerable build for four weeks while I admired its uptime.

That is worse, and it is worse in a way nobody would ever have noticed.

The argument is narrower than that. The automation is production. It runs as root, it holds the socket, it restarts your containers, and it does all of that at four in the morning when nobody is watching.

It has earned the same suspicion you would give anything else with those privileges, and it does not get it, because you have filed it in your head under maintenance instead of under infrastructure.

Go and look at what yours can actually see. Not what you configured it to see. What it tells you it can see.