essay

An LLM reviewing an LLM is not a safety bound

A second AI catching the first one's mistakes makes errors rarer. It never makes them impossible, because the two fail on the same inputs. Software engineers measured this in 1986, with human programmers. So an action that cannot be undone needs a check that is not a model at all.

Robert Gallas · August 2026

Agents are moving from answering questions to taking actions: paying the invoices in a mailbox, updating records, calling other systems. That raises a question that did not matter before: what stops an action that should not happen?

The default answer is another model. Put a reviewer in front of the actor — a guardrail, a critic, a judge model — and let it veto. Measured before and after, the numbers usually improve. But for actions that cannot be undone, this essay makes a stronger claim: if the judge is the thing you rely on to let the action through, it makes the system worse.

A smaller error rate is not a guarantee

A reviewer that catches most bad actions turns a small failure rate into a smaller one. Nothing in that setup can ever say this cannot happen. It can only say this happens less often. Engineers call the first kind of statement a bound, and a bound is what you want before something irreversible runs.

A question worth sitting with for a moment: is a calculator that is wrong one percent of the time better than a calculator that is wrong five percent of the time? The answer depends entirely on what you think a calculator is for.

And smaller is the best case, because agents retry. A refused action comes back reworded, and every retry is a new chance to slip through. A reviewer that misses five percent of the time will, more often than not, approve a persistent agent within twenty tries. For mistakes you can undo, that is tolerable. When the payment has left the bank or the dose has been given, the useful question is what the system cannot do at all, and only something that does not work by chance can answer it.

The two models fail together

Two checks are better than one only if they fail at different times, and that assumption has been tested. In 1986, Knight and Leveson had twenty-seven separately written versions of the same program run against a million test cases. Each version was reliable on its own. When they failed, they failed together, on the same inputs, far more often than chance predicts. The hard cases of a problem are hard for everyone.

Two AI models have far more in common than those programmers did: much of the same training text, the same basic design, sometimes literally the same model, and always the same input. Recent measurements across today's models show their errors line up well beyond chance, and more strongly as the models get better.

The shared input is the worst part. A text written to trick a model is read by both of them: the actor was persuaded by it, and the judge reads the same words in much the same way. An attacker keeps trying until something works, so the version that finally works is, by definition, one that got past the judge too. And no attacker is needed: being wrong sometimes is how these models normally operate, and a wrong answer arrives sounding exactly as confident as a right one.

The usual reply is that judging is easier than doing. That is true exactly where the check is mechanical: an amount against a limit, a destination against a list. But a mechanical check does not need a model at all. Where the check takes judgment, the judge is doing the same work as the actor, on the same text.

What a guarantee looks like

Companies have always handed real work to people they could not fully check. A clerk who pays invoices works under a spending limit, an approval threshold, a second signature, a ledger. The procedure limits the damage, and it does not depend on the clerk's judgment being sound.

Applied to an agent: the model's output is a proposal — pay this supplier, this amount, to this account — and ordinary code decides whether it runs. No model takes part in that decision, and the rules can be read, tested, and reviewed by someone who does not trust you.

What you get is a different kind of statement. "No payment above ten thousand without a second approver" is true every time, including against crafted attacks, because the check does not read the attacker's argument. It reads a number and compares it.

A persuasive document can still make the agent propose anything at all. It cannot raise the limit, because the agent was never holding that decision. And retrying stops working: against a reviewer, every retry is a fresh chance at a miss; against a rule that reads the same number, every retry gets the same answer.

What it does not solve

A fixed rule reads records, not intentions. It catches a payment that matches no open order; it misses the invoice that is wrong in a way nobody has recorded yet, exactly as a human clerk would, and with the same remedy: record the dispute, and the procedure holds the payment. A fixed rule does not remove failure either; it moves it. The failure stops being the model was talked into something and becomes the rule was wrong, and a wrong rule can be reviewed before it runs, tested, and tracked in version control like any other code. Model-based filters are still worth having behind the rule, as extra layers. The one thing they cannot be is the reason an irreversible action was allowed to happen.

None of this makes agents safe. It turns an open-ended risk into a limited one you can reason about, which is the trade engineering has always made wherever unreliable parts hold real authority.

---

The part we would most like to see refuted is the correlated failure claim. If there is a way to combine two models into a genuine guarantee rather than a smaller rate, we have not found it, and we would rather be shown it than keep asserting it. Argue with it here, or by email. The companion essays: Gate the enabling change, on where the rules belong, and A familiar procedure, on how old this machinery is.

---