essay

An LLM reviewing an LLM is not a safety bound

A second model in front of the first lowers the error rate at best, and it never gives you a bound: the two fail on the same inputs. Software safety engineering learned this in 1986, from human programmers.

Robert Gallas · 2 August 2026

Agents are moving from answering questions to taking actions. An agent reads a mailbox and pays the invoices in it, updates records, calls internal APIs. Once that happens, someone has to answer a question that did not previously matter: what stops an action that should not happen?

The usual answer is another model. Put a reviewer in front of the actor, have it inspect the proposed action, and let it veto. Guardrail classifiers, critic models, LLM-as-judge, constitutional checks: the pattern has many names and it is now the default. A reviewer catches a large share of bad actions, and measured before and after, the numbers improve.

The numbers improving is not the same as the system being safe. Those are different properties, and for actions that move money or change a medical record, only one of them counts. For those actions this essay makes a stronger claim: a judge model is not just insufficient there. Deployed as the thing that makes an irreversible action permissible, it makes the system worse.

A rate is not a bound

Suppose the actor proposes a wrong action some fraction of the time, and the reviewer catches most of those. The result is a smaller fraction. It is still a fraction. Nothing in the arrangement produces a statement of the form this cannot happen, only statements of the form this happens less often than it used to.

A small question, worth an honest answer before reading on: is a calculator that is wrong one percent of the time better than a calculator that is wrong five percent of the time? Decide what a calculator is for, then answer.

And smaller is the favourable case. A veto on its own cannot make the executed-error rate worse, but the system built around the veto can. Agents retry: a refused action comes back reworded, and each attempt is a fresh draw against the reviewer's miss rate. By the same arithmetic that justified stacking models in the first place, a reviewer that misses five percent of the time per attempt lets a persistent actor through with better-than-even odds inside twenty tries — caught ninety-five percent of the time becomes approved eventually. A reviewer that edits proposals instead of just refusing them is a second generator, a source of wrong actions the actor never proposed. And a reviewer believed to work invites wider autonomy and fewer human checks than anyone would grant the bare model, so total exposure can rise while the per-action rate falls. Retrying until approval is how agent loops already behave.

For most software the distinction between a smaller rate and a bound is academic, because most errors are recoverable. You notice it and you fix it. It stops being academic when the action is not recoverable: the payment left the bank, the dose was administered, the records were disclosed. There the useful question is not how often the system is wrong. It is what the system cannot do, for every input, including inputs nobody has thought of yet.

That question has an answer only if something in the path is not probabilistic.

The multiplication assumes independence, and independence does not hold

The intuition behind stacking models is that two independent checks multiply: if each fails one time in a hundred, both fail one time in ten thousand. This is sound arithmetic and the wrong model of the situation, because it depends entirely on the failures being independent.

Software safety engineering has already tested that assumption, carefully, and it did not survive. In 1986 John Knight and Nancy Leveson published an experiment on N-version programming, the practice of running several independently written implementations and voting on their outputs. Twenty-seven versions were written from the same specification by programmers at two universities, then run against a million test cases. The individual versions were reliable. The problem was that when they failed, they failed together, far more often than the independence model predicted. The assumption was rejected statistically.

The cause transfers directly. The versions shared nothing except the specification and therefore the problem, and the hard parts of a problem are hard for everyone. The correlation came from the work itself, not from programmers copying each other.

Two language models share far more: overlapping training corpora, the same architecture, often the same model family, sometimes literally the same weights behind a different prompt — and above all, the same input. If correlation emerged from a shared problem alone, it does not weaken when the two processes also share their training and their reading material. That is why the age of the result does not date the argument: it is not a finding about neural networks but about similar processes working the same hard case, which is the position a judge model is placed in.

The shared input is the dominant common mode

The reviewer has to see the material in order to review it. If a document contains text crafted to make a model act a certain way, that text is now in front of both models. The actor was persuaded by it. There is no particular reason to expect the reviewer, reading the same words with the same machinery, to be immune.

The adversarial case is worse still. An attacker is not drawing samples from the distribution you measured; an attacker iterates until something works, so the input that eventually succeeds is, by construction, one that gets past the reviewer as well. Error rates observed on ordinary traffic say little about traffic from someone who is trying.

None of this requires an attacker, though. A model produces each token by sampling from a probability distribution, so wrong output is a normal operating mode. Hallucinated details, misreadings, and losing the thread of a long task are surface forms of that one mechanism. An agent with no adversary anywhere will still occasionally propose paying a vendor that is not owed anything. When it does, the proposal arrives with the same fluency and the same confidence as a correct one, which is what the reviewer has to work with.

The usual reply is that judging is easier than generating, so the reviewer need not fail where the actor does. Sometimes that is true; the question is where. The check is easier when it is mechanical: is the amount under the limit, is the destination on the list, has this order line already been paid. Checks of that kind do not need a model at all; written as ordinary code they are exact rather than probably right. The check is not easier when it is interpretive: whether a sentence was an instruction or data, whether an invoice is genuine. There the reviewer does the same work as the actor, on the same text, with the same kind of machinery. Where the asymmetry is real, it is an argument for taking the check out of the model, not for adding a second one.

Where a probabilistic check is the right tool

The claim here is narrow, and what it does not say matters. Layered mitigation is real engineering: a classifier that catches most prompt-injection attempts, or a reviewer that flags suspicious actions for a human, is worth having behind a bound, as one more filter. What it cannot be is the thing that makes an irreversible action permissible. Reducing the rate of bad events is a legitimate goal and most of security works this way.

The narrow claim is that a probabilistic check reduces a rate and does not establish an invariant. A useful rule of thumb: if the failure is recoverable, rate reduction is a reasonable answer. If the failure is not recoverable, you need a property that holds every time, and no arrangement of models will give you one.

What a bound looks like

The alternative predates software. Organizations have always delegated real authority to actors they cannot fully verify, and they manage it the same way every time. A clerk who pays invoices works under a spending limit, an approval threshold above it, a second signature above that, a ledger, and an audit trail. Nobody attempts to make the clerk incapable of error first. The procedure is what bounds the damage, and the procedure does not depend on the clerk's judgment being sound.

Applied to an agent, that means the model's output is a proposal rather than a decision. The proposal is expressed in a structured form with typed fields, and ordinary code decides whether it executes: the mechanical checks from earlier, plus whether the action requires a human first. None of those checks involve a model. They are the same kind of code as the rest of your system, and they can be read, tested, and reviewed by someone who does not trust you.

The property you get is categorical rather than statistical. "No payment above ten thousand without a second approver" is true for every input, including the crafted ones, because the check does not read the input's argument. It reads a number and compares it.

A persuasive document can still convince the agent to propose anything at all. It cannot raise the limit, because the limit is not something the agent is asked about. The agent was never holding that decision. Retrying does not help either: against a reviewer, every retry is a fresh chance at a miss; against a rule that reads the same number, every retry reaches the same verdict.

What determinism does not solve

Being honest about the limits matters more than the argument, because the limits are where this approach gets sold past its capability.

A deterministic check cannot read intent. It can read records, and much of what looks like intent is recorded somewhere: an invoice is owed because an open purchase order says so, a dose is due because a prescription says so. Checking a proposed action against the system of record is still deterministic, and it catches the common ways an agent pays something that is not owed: no matching order, wrong amount, already paid. The check runs on the organization's recorded knowledge, not on the model's judgment, which is why it holds even when the model is confused.

The residual case is wrongness that no system has recorded. The delivery arrived broken, nobody has entered the dispute yet, and the invoice still matches its order; an agent that pays it is wrong in a way no rule can see. This is not a weakness the gateway introduces. A human clerk fails the same way, for the same reason: clerks also pay what the records say is owed. The remedy is the same in both cases — record the dispute, and the procedure holds the payment. Where recording cannot be fast enough, the policy decides which actions wait for a person, and the audit record carries the identifiers needed to find and reverse what went through. Detection and reversal are weaker than prevention, and should be described as weaker.

The checks are also only as good as what someone wrote. Determinism does not eliminate the failure, it relocates it, from the model was talked into something to the rule was wrong. That is a real trade and the second failure is not free. It is, however, a failure you can review before it happens, test, put under version control, and hand to an auditor. The first kind offers none of that.

So the honest summary is not that deterministic enforcement makes agents safe. It is that it converts an open-ended risk into a bounded one you can reason about, which is the trade that engineering has accepted in every other domain where unreliable components hold real authority.

---

I have been building one of these, called Stonefold: a checkpoint that sits between an agent and the systems it can touch, with the policy in a small declarative file and no language model in the enforcement path. It is Apache-2.0, the specification is an evolving draft, and the implementation is prototype grade. It is not the point of this essay, and the argument stands or falls without it.

The part I would most like to see refuted is the correlated failure claim. If there is a construction in which two models give a genuine bound rather than a smaller rate, I have not found it, and I would rather be shown it than keep asserting it. Argue with it here, or by email.

---