Nothing Broke, and That Was the Problem

Nothing Broke, and That Was the Problem

Most of the bugs I write about announced themselves. The site went down, a certificate went wrong, something threw a 500 and I went looking.

That is the normal shape of the work. Something visibly breaks, you follow the smell, you find the cause four layers from the symptom, and you feel clever for an afternoon.

This one did not do that. Everything worked exactly as designed, for two days, and the whole time a folder of my own private notes was sitting at a public URL waiting for anyone who thought to guess it.

I found it because I went looking for it. Not because anything told me.

The thing I built to show strangers

A while back I set up a small host for sales mocks. The idea is simple enough.

When I want to show a prospective client what their site could look like, a PDF is weak and a Figma link is worse, so I build the actual thing and put it on a subdomain of my own domain. They open a link on their phone and see a real site.

The mechanics are deliberately boring. Every mock lives in a folder, one folder per prospect, and everything in that folder gets served at that prospect's subdomain.

One nginx container, a static tree, no application code. The kind of setup you can hold entirely in your head, which is the only kind I want to be responsible for at eleven at night.

The part I did not think hard enough about is what else lives in those folders. A mock is not just an index.html.

When I put a pitch together there is research behind it: an audit of what their current site gets wrong, notes on their competitors, the outreach sequence I planned, and the pricing I was going to quote.

That material sits in the bundle because that is where the work happens.

All of it, in other words, was in a directory whose defining property was that its contents get published.

Three lines that all looked like the same promise

I had thought about this. That is the uncomfortable part. My .dockerignore was not empty, and it did not look careless:

scripts
README.md
*.md

Three entries. One of them names a markdown file directly. One of them is a wildcard over every markdown file.

Read that in a hurry and it says: markdown does not ship. I read it in a hurry many times, because I had written it, and you do not audit your own settled decisions.

It was not doing what I thought. README.md excluded exactly one file at the top of the build context.

And *.md, the line that looks like the comprehensive one, the line that looks like it closes the whole category, was excluding almost nothing at all.

A star does not cross a slash

Docker's ignore file is not Git's ignore file. They sit next to each other in the same repository root, they use the same syntax, and they disagree about what that syntax means.

Docker matches patterns using Go's path matcher, and in that matcher the star is not allowed to cross a directory separator.

So *.md means "markdown files sitting directly in the build context root" and nothing else. Docker's own documentation states this plainly, in a note explaining that markdown files "under subdirectories are still included".

To reach deeper you have to write **/*.md, where the double star is the thing that walks down the tree.

Every file I actually cared about was nested one level down, inside a per-prospect folder. Not one of them was ever excluded.

Now hold that next to .gitignore, where the rules are inverted. Git treats a pattern with no slash in it as matching at any depth, so the same three characters recursively catch every markdown file in the repository.

If you want Git's version to be root-only, you have to add a leading slash to narrow it.

So the same literal string, in two dotfiles in the same directory, means opposite things. In one it is recursive by default and you restrict it. In the other it is shallow by default and you widen it.

Neither file will tell you which one you are editing.

I have been doing this for over twenty years and I did not know that. I would have told you, confidently, that *.md meant all the markdown.

I would have been wrong in one of the two files and right in the other, and I would have had no idea which.

The failure mode is silence

Here is what makes this class of bug different from the ones I usually write about. When I broke TLS on my own site a while back, the internet told me immediately.

Cloudflare threw a 526, the site was unreachable, and the feedback loop was measured in seconds even though the actual cause was four layers downstream. Painful, but honest. The system was screaming.

A deny rule that under-matches screams nothing. Every mock rendered perfectly. Every prospect who opened a link saw exactly what I wanted them to see.

No error, no alert, no log line, no slow page, no failing healthcheck. The system had no opinion about the situation at all, because from its point of view there was no situation: it was asked to serve a directory and it served the directory.

There is nothing to notice. That is the entire problem. A rule that fails loudly gets fixed on the day it fails. A rule that fails silently gets promoted to an assumption, and assumptions do not get re-read.

And I should be honest that this is not the first time Docker's tooling has quietly told me a rule was in force when it was not.

I have written before about publishing a container port straight through a firewall that swears the port is closed, where ufw status and reality disagree completely and neither one flinches.

Different mechanism, same shape. The tool reports a state. The state is not the truth. Nothing in between raises its hand.

Planting something I wanted to lose

I did not find this by reading the config again. Reading it again is how it survived in the first place, because on every re-read I recognised my own intent rather than the file's behaviour.

I found it by testing the exclusion the way you would test any other claim. I put a file called SECRET.md into a nested folder inside the mock tree, specifically so I could go looking for it afterwards.

Then I built the image and checked whether it was in there.

It was baked right in.

That is a five minute experiment and it inverted a belief I had held for years. It also generalises further than I would like.

Any rule whose job is to prevent something has this property: it produces no evidence when it works, and no evidence when it fails. Backups, firewall rules, ignore files, permission checks, retention policies.

The only way to know is to make the bad thing happen deliberately and confirm it got caught.

I now do not trust an exclusion I have not watched reject something.

Two locks, because one of them is only a belief

The fix has two layers, and it has two layers on purpose. At build time, the ignore file got patterns that actually descend: **/docs/, **/*.md, **/*.pdf.

That is the real fix, because material that never enters the image cannot be served from it.

At serve time, nginx refuses the path outright:

location ^~ /docs/ {
    return 404;
}

That ^~ is doing specific work. nginx normally checks prefix locations, remembers the longest match, then goes off and evaluates the regex locations, which can override it.

The ^~ modifier tells nginx to stop there: if this is the longest matching prefix, do not check regexes at all. So this rule wins even against a regex block that would otherwise happily serve an image out of that folder.

The second layer is not there because I think the first one is broken. It is there because I thought the first one was working once already, and I was wrong for two days without noticing.

The second lock exists to catch the failure of my own confidence, which is the failure I have the least visibility into.

The fix I did not make

There is a stronger position than mine, and I want to state it properly rather than knock down a weak version of it.

The argument goes: you should not be filtering private material out of a public artifact at all. The mistake was not the pattern, it was building a directory that mixes the thing you publish with the thing you must never publish.

Get the layout right and the rule becomes unnecessary. No filter, no filter bug.

That is correct, and I did not do it. The reason is practical: the bundle is the unit of work.

When I am putting a pitch together, the research and the mock are the same task, and splitting them across two trees means two places to look, two things to keep in sync, and a real chance I lose the notes entirely.

I chose the ergonomics and paid for them with a rule I then had to get right.

What I would say in my defence is that the two answers are not exclusive, and the person who separates the trees still needs the canary test. Their build could copy the wrong directory, their deploy could resolve a symlink, their nginx could have a root one level too high.

The layout removes one specific failure. It does not remove the category, and the category is what actually bites, because the category has no symptoms.

Nobody sends you an alert about the door you left open

None of this is exotic, which is exactly why it is worth writing down.

Truffle Security scanned the top million websites a few years back and found roughly 4,500 of them serving a publicly readable .git directory. When they pulled credentials out of those repositories, most of the GitHub tokens they recovered carried admin rights.

GitGuardian scanned fifteen million public container images and turned up around a hundred thousand valid secrets across a hundred and seventy thousand images.

Flare, looking at a single month of uploads, found ten thousand images carrying exposed credentials, and reported that where the secret was later removed, three quarters of the time nobody rotated the underlying credential.

Those numbers do not describe careless people. They describe competent people whose exclusion rule was narrower than they believed, in a system that had no way to tell them, working on something that was functioning perfectly the entire time.

I do not know whether anyone ever opened one of those files on my host. I know the door was open for about two days, I know exactly which pattern held it open, and I know that nothing in my stack was ever going to mention it.

The mocks are retired now, and the two locks are on the door for the next set.

The bugs that cost me an afternoon are the ones I enjoy, in retrospect. The ones that cost me nothing at all, and would have gone on costing me nothing right up until the moment they cost me a client's trust, are the ones I have learned to go hunting for on a quiet day.

Go plant a file you would hate to publish, and see where it ends up.