The Second Agent Is Not a Second Source

The Second Agent Is Not a Second Source

I run a small fleet of research agents three times a week. They go out, read pages, and come back with claims and URLs, and I write from what survives. It is a good arrangement and I have been quietly proud of it, mostly because of the part where I check their work.

At the end of July, two of them came back with the same claim. Cloudflare, they said, was relaunching Pay Per Crawl as Pay Per Use. Both cited a Cloudflare blog post. Two independently briefed agents, same finding, same source, and the source was the vendor's own site rather than some aggregator.

So I fetched the page. The phrase "Pay Per Use" is not on it. Not once.

I wrote a correction into my own notes that afternoon, in slightly self-satisfied language, telling my future self not to repeat that claim from that URL. Two agents had pattern-matched a plausible product story onto a real link, and I had caught it before it reached anything with my name on it.

The claim was true and the citation was not

Preparing this piece, I went back to check my own correction, because writing about a mistake is a good reason to make sure you understood it.

Cloudflare renamed it. There is a post on their blog dated the 1st of July 2026 with a section heading that reads "From Pay Per Crawl to Pay Per Use", and a sentence about shaping the one into the other. It went up four weeks before my scan ran. My agents were right about the world and wrong about the page, and the page they were wrong about was the one I checked.

That is a worse failure than the one I thought I had found, and it is worse in an interesting direction. A fabricated claim with a bad citation dies the moment you open the link.

A true claim with a bad citation survives every check you are likely to run, because checking the claim confirms it and checking the citation only tells you to drop it. My verification step had exactly one move available, and that move deleted a fact that was real, public, and four weeks old.

A smaller version of this bit me the same fortnight. I asked a fetching tool which OAuth scope guards a planets endpoint in EVE Online's API, and it told me, confidently, esi-planets.read.v1. I have since pulled the raw spec down and grepped it. That scope does not exist.

The real one is esi-planets.manage_planets.v1, and it was already sitting correctly in my own shipped code. The invented name did not read as a bad answer. It read as a bug in code that was fine. These tools answer by running a small fast model over the page, so on anything machine-readable you get a paraphrase, and an identifier is the one thing a paraphrase cannot approximate.

Two agents are not two sources

I had been running on an assumption I never wrote down: that two agents briefed separately, given different prompts, are two witnesses. Concurrence meant corroboration. It is such a natural thing to believe that I never noticed I believed it.

They share training priors. They fail the same way on the same inputs at the same time, and when they do, the agreement I am reading as confirmation is one opinion I paid for twice.

Somebody has now measured this properly. A paper from May, "Nine Judges, Two Effective Votes", put nine frontier models from seven different model families on the same evaluation task and asked how much independent information the panel actually carried. The answer is in the title.

Nine judges were worth about two independent votes, and roughly three-quarters of the panel's nominal independence was gone because the models made the same mistakes on the same items. The panel's accuracy came in 8 to 22 percentage points below what genuinely independent voting would have produced, and the best single judge matched or beat the whole panel in every condition tested.

The part I keep returning to is that better aggregation did not rescue it. Established methods closed at most 11% of the gap even when handed the correct answers. The bottleneck was the correlation, not the arithmetic on top of it.

My instinctive fix, before I read any of this, would have been to use different models. Seven model families is what that fix looks like when you actually run it, and it bought about two votes. A paper from last year with the useful title "Rethinking Mixture-of-Agents" points the same way: aggregating outputs from the single best model beat mixing several different ones by 6.6% on one benchmark and 3.8% across several others, because mixing in weaker models drags the average quality down faster than the diversity buys anything back.

Nominal diversity is not decorrelation. I had been treating them as the same word.

The phantom test file

Priors are the channel everybody at least suspects. The one that got me last week is more physical than that. I had several agents working against one checkout, and one of them needed to settle a rendering question, so it wrote a throwaway test file, ran it, and deleted it. Exactly what I would have done.

A second agent, auditing the same checkout, found that file mid-run, read its assertions as evidence of what the codebase actually does, tried to execute it, discovered it had vanished, and spent a paragraph of its report reasoning carefully about a thing that no longer existed. A third one hit it too.

All three were right to flag it. None of them could tell it apart from a stray file a human had left lying around, because there is nothing to tell apart. It was a real file with real assertions in it. It was also one git add -A away from being committed, which is a different article and a worse afternoon.

The agents were independent in the way I had designed for. They had separate prompts, separate context, no visibility into each other's reasoning. They were sharing a filesystem, and a filesystem is a communication channel whether or not you meant it as one.

The vendors ship the shared disk on purpose

I assumed this was my own untidiness until Black Hat, where researchers took apart the multi-pass agent pattern that OpenAI documents, builds on, and runs on its own repository.

It is the pattern you have probably been told to build, and it looks careful. Pass one reads the untrusted input and controls nothing directly. A deterministic check sits between the passes. Pass two only ever sees a validated value, so pass two is the one you trust with the token and the write permission.

Pass one can write to the workspace. AGENTS.md is the instruction file the agent loads off disk on every invocation and treats as authoritative, and it sits in that workspace, unprotected, while the neighbouring metadata directories are locked read-only.

So pass one writes the instructions, and pass two, being a fresh process in the same directory, picks them up as its own. The isolation between the passes is real and it is entirely in the data path. Underneath it, they share a disk.

The researchers' sharpest observation is that failing the validation check is what launches the second pass. The check does not stand between the attacker and the prize. It is the trigger.

None of this requires anybody to be attacking me. A paper from April called "No Attacker Needed" measured shared-state agents and found contamination rates of 57 to 71% from benign interactions alone. Just agents working normally, in the same place, leaving things where the next one finds them.

Independence is not ignorance

The third channel is the one I built myself, and it is the one I am least happy about. When I hand a claim to an agent whose only job is to refute it, I withhold my reasoning, because that is the whole point of the seat. Somewhere along the way I started withholding more than my reasoning, on the theory that less context meant a cleaner read.

So I gave a refuter the observations but not the samples, and it constructed a detailed alternative explanation out of arithmetic on the numbers I had given it. The samples I had withheld refuted it in one line.

Another time I withheld the document under review itself, to keep the seat blind, and got back a confident finding that the claim was unsupported by any of the material it had been shown. That was true. It was also a fact about my prompt rather than about the claim.

Both objections cost a full agent pass and an adjudication round to dismiss. Neither was stupid. Both were built precisely on the gap I had left.

Withholding my conclusions is what makes a refuter independent. Withholding the evidence just makes it uninformed, and an uninformed refuter builds confident counterarguments out of the hole where the data should have been.

The agreement I trust more than the others

I do not want to leave this sounding like the fan-out is worthless, because a fortnight ago it caught something I would not have. Three agents, given different lenses and none of my reasoning, each independently reported the same specific defect: a column I was proposing to display is empty on half the paths that can produce it. Not one of them had been asked about it. One file read confirmed it.

That is a different object from three agents agreeing with me, and the distinction is the practical half of all this. Shared priors explain several models reproducing the same plausible story. They do not explain three separately scoped reads landing on one specific mechanism nobody pointed them at.

So the discount applies to concurrence on my own conclusion. Convergence on something I never claimed now goes to the top of the queue.

The best evidence for multi-agent setups points the same way. The 2024 debate result, where debate lifted non-expert model accuracy from a 48% baseline to 76%, works because the debaters hold information the judge does not and are pushed to surface it.

The gain comes from the asymmetry, not from the headcount. The authors do not claim it generalises to symmetric agents checking work that all of them can see equally well, which is the arrangement I actually had.

My own checker said eight real papers did not exist

While writing this, I went to verify the papers above against arXiv's API rather than trusting the agent that found them, because that is supposedly the lesson.

My check came back clean and damning. Eight papers, none of them found, including a well-known one from 2018 that I was fairly sure existed. That last one was the tell. The API had answered 301, my command had not been told to follow redirects, and my verifier had been confidently reporting nothing where there was something.

Every one of those eight is real. I have the titles, the dates, the authors, and the numbers off the raw abstracts.

But for about ninety seconds I had a clean-looking negative result produced entirely by my own tooling, and if I had been checking a claim I already doubted rather than eight I expected to find, I would have believed it. That is the same failure as the Cloudflare correction, running in the other direction, on the same afternoon.

A checker that wrongly says no is not safer than one that wrongly says yes. It just fails somewhere you are less likely to look, because a deletion leaves nothing behind to audit.

What I actually changed

Scratch output goes outside the repo now, and when a fan-out includes an agent that writes to disk, the agents that only read are told so and given the filename pattern to discount. I stage explicit paths while anything parallel is live.

Anything load-bearing that lives in a spec or a config gets fetched raw and grepped, never summarized. Identifiers especially. If I am going to print a scope name in an article about invented scope names, I am going to have grepped for it myself, which I did, twice.

Refuters get the artifact under review and every observation I hold. They do not get my conclusions.

And when two agents agree with me, that goes to the front of the checking queue rather than the back, which is the exact inverse of how it feels. Agreement is what makes a claim feel safe to skip, and feeling safe to skip is the entire mechanism by which the wrong thing gets published.

The thing I have not solved is the one that started all this. My correction was right about the citation and wrong about the world, and it sat in my notes for a fortnight telling my future self that a true thing was not true.

I have fixed that particular line. I have not fixed the general case, because I have no step whose job is to ask whether the claim I just deleted might have been right for reasons its citation failed to mention.

Nine judges were worth two votes. I would settle for one that tells me when it has thrown something away.