There Are No Real Referees in AI Security

There are 342 automated tests sitting in my repo. I didn’t write them — I barely have a working concept of tests. An AI wrote them. Same for the security design I only noticed later: which data columns must never appear in a public-facing form, remote-control commands parsed against a whitelist and never executed raw. None of that was my thinking. When I was once asked who designed the trust boundaries, my answer was two letters: AI.

The same system writes the exam and grades it. I didn’t think much of this until I watched a recent podcast episode and realized the entire industry is soaking in the same structure.

The guests were the two founders of Gray Swan, the company that runs pre-release attack testing for just about every frontier lab — they did the indirect-prompt-injection testing on Claude Mythos before launch, and they closed a $40M Series A last month. Co-founder Zico Kolter has a second job that makes this interesting: he sits on OpenAI’s board and chairs its Safety and Security Committee — the panel with the power to halt an OpenAI model release.

So one person is simultaneously the industry’s most important safety referee and the co-founder of a company whose business improves as models get less secure. That’s not a conspiracy theory; it’s on his Wikipedia page. They’re candid about the business model too: customers tend to show up after something has already gone wrong. Even the company name is honest — a black swan is the event nobody sees coming; a grey swan is the one you watch flying toward you. They’re waiting for the first one to land.

Should he recuse? I think yes. Will recusal ever actually happen? No — not in the early phase, and not in the late one. The early reason is simple: only a handful of people truly understand attacks. Kolter and his co-founder Matt Fredrikson spent over a decade at CMU on adversarial machine learning; the 2023 universal jailbreak paper that broke ChatGPT, Claude, and Bard in one shot came from them and their students. If you want a referee who understands the game, you can’t route around these people — route around them and your referee can’t read the field.

And later? The conflict of interest doesn’t disappear; it goes underground. Once an industry matures, influence stops needing a job title — it becomes advisory seats, LP checks, revolving doors, nothing you can look up. Cryptography already ran this play: the NSA worked its way deep into America’s official crypto standards, ending up sole editor of one random-number standard; the Snowden files all but confirmed the backdoor in it; and RSA — a security company — was reported to have taken $10 million from the NSA to make the backdoored algorithm its product default. By that measure, today is the honest phase: at least the conflicts are sitting on the table.

What about institutions, then? The episode conveniently walked through both candidates.

First, compliance. Fredrikson trashed SOC 2 himself: it’s a product of accountants, not security experts — and he’s literally right, it’s an AICPA framework audited by CPAs. In practice SOC 2 stopped being evidence of security long ago; it’s a procurement passport, a stamping business. AI-security compliance will most likely walk the same road.

Second, insurance. The founders put real hope in AI insurance: insurers pay actual money when things break, their incentives point opposite to vendors’, so they have reason to make risk assessment rigorous instead of ceremonial. It sounds airtight. But in 2008 the identical logic died on the beach: rating agencies also supposedly lived on credibility, yet they took the issuers’ money and stamped AAA on subprime all the way down. Opposed incentives don’t prevent capture; they just make capture take one extra corner. My call: AI insurance ends up in the same place.

So the conclusion is bleak: there are no real referees in AI security. Not now, and not later.

But the conclusion has an exit, hidden in one technical detail from the episode. Frontier models, they said, are terrible at red-teaming — their own safety training makes them refuse to attack other models, so attack models have to be trained separately, as a different system from defense. I’ve hit the same structure in my own project: a data bug, three or four diagnostic tools written by the AI, no root cause found — until I switched to a different AI, which cracked it quickly. Not because the new one was smarter; because it didn’t carry the sunk cost of the first one’s wrong assumptions.

A model can’t red-team itself. An AI can’t dig through its own bad assumptions. A human can’t inspect the blind spots they never click on. Nobody gets to be their own referee — and that holds at all three levels: the individual, the system, the industry.

If real referees don’t exist, the move is not to keep searching for a neutral party — neutral parties either don’t understand the game, or don’t stay neutral for long. The move is to keep the self-interests pointed in opposite directions on purpose: whoever writes the exam shouldn’t grade it; attack and defense get trained as separate systems; whoever does the assessment should lose real money when they’re wrong. Every party can have an agenda, as long as the agendas collide.

The same standard applies right back to my own repo. In my pipeline, an AI reads external content, has access to all my data, and can publish straight to the web — Simon Willison calls this the lethal trifecta, and with all three assembled, a prompt-injection attack has everything it needs. Meanwhile my exam-writer, exam-taker, and grader are still one system. What I can actually do is not find a “trustworthy guard,” but hand one of those legs to a separate system that shares no interest with the first. That doesn’t solve the problem. It postpones the day it happens.

Until the grey swan lands, that’s about all any of us can do.