essay
Gate the enabling change
Companies put approvals on the actions that sound dangerous. The expensive failures usually start earlier, in a small administrative change nobody was watching. Finance learned this years ago. And the check that works is plain software, because the AI model underneath keeps changing.
When an AI agent needs oversight, the first design most people reach for puts the approvals on the actions that sound dangerous: an approval on every payment, a sign-off before anything is deleted, a review step before the queue is cleared. It feels so obvious that nobody questions it. The payment is where the money leaves, so the payment is where the person looks.
An attacker sees the system differently. A guarded action is hard to attack directly, and it often does not need to be attacked at all, because whether it looks correct depends on records that some other, unguarded action can change. Change a supplier's bank account, and every later payment to the attacker passes every check that looks only at the payment and its documents: real supplier, real invoice, real order, right amount, within limits. By the time the payment happens, the fraud is already done. The person checking the payment sees documents that all agree with each other, because the record that would have given it away was changed a week earlier. It is the same trick as redirecting someone's mail: the thief does not forge anything afterwards, because everything now arrives at the address they chose.
Finance already learned this
None of this is news to a finance department. Payment-redirection fraud taught finance exactly this, over decades, by losing money. The standard control they settled on is the one this essay is about: when a supplier asks to change its bank details, someone picks up the phone and checks, while the payments keep running. A finance controls owner who reads the paragraph above and thinks we already do this is right, and that is the point.
And finance is not the only field that learned something like it. Where a domain felt real pressure, some version of the habit usually followed: production changes that go through an approval process, code that cannot ship without a second person's review, record merges treated as work for a person rather than a script. An expert in any of these fields can say we already do this, and they are often right.
Now look at what those controls are usually made of. Most are human process: a meeting, a review queue, a named approver clicking a button. Stricter, machine-enforced versions exist in places, but as options that only protect where someone has switched them on. An AI agent does not sit in the meeting and never sees the review queue. It acts through APIs, at machine speed, and that is exactly the surface the human-process controls do not reach. In the system we tested, the cloud provider's delete-protection flag could be switched off by any identity with ordinary configuration rights, with no approval and no notification.
AI agents change who faces the pressure. An agent can do the volume of work of a whole back office, and it takes instructions from whatever it reads, so every email, ticket, and document it touches could be an attack. Both of these arrive together, wherever an agent is put to work. And the volume is what makes the obvious control placement useless: a person asked to review hundreds of correct-looking actions a month stops really reviewing them, and a control that has quietly become a formality is worse than no control, because it looks like one. There is only so much honest attention a person can give. Spend it on the request that comes a few times a year and that the fraud cannot skip.
What we tested
We tested this in three systems built to behave like real ones, each driven by a real language model, and we let the failures happen before writing any control; the use cases describe each one, including what the control will not do.
In the payments system, whether the agent fell for a planted email depended on which model was driving; the rule on the bank-detail change made the outcome the same either way.
In the cloud system we built both designs and ran them against the same scenarios: an approval on deletion itself, and an approval only on switching the protection flag off. Losses were zero under both. Of the four scenarios in the set, one contained a legitimate deletion; the design that gated deletion blocked it, and the design that gated the enabling change let it through. That is the only head-to-head comparison we have run so far.
In the clinical system the enabling change is the record merge. Merges are usually legitimate, which is exactly why a wrong one hides so well: after a merge, everything downstream looks correct, because the records now agree. With no checkpoint in place, the agent's merge simply ran, and nobody was asked. With one, it waited for a person.
The check cannot live in the model
Look again at the payments result. Two models were given the same system, the same documents, and the same instructions. One never attempted the fraudulent change; the other attempted it on every run. Nothing else was different. Only the model.
A control that lives in the model inherits everything you do not know about the model. Its behavior can shift when the vendor ships a new version. It can be swapped for a cheaper model when the bill arrives. The same model acts differently under a different prompt or harness. It was trained by someone else, on data you cannot practically check. And a company running several agents may be running several models at once, from different sources, each changing on its own schedule. None of this is a reason to avoid good models. It is a reason not to let the safety of a payment depend on something that changes underneath you.
So the check on the enabling change has to be plain software: a written rule, applied the same way every time, sitting outside the model, where nothing the model reads and no update the model receives can move it. That is what made the payments outcome the same under both models. And you know today exactly how that rule will behave next year, which you cannot say about anything inside a model.
What that software has to do
It has to see the boring actions. There is no shortcut where you route only the dangerous systems through it: the supplier table looked like plumbing until it turned out to be the whole fraud, and the protection flag looked like a provider detail. You cannot know in advance which boring system will turn out to matter, so covering only some of them is a bet that you guessed right. In practice the agent holds no credentials of its own, and the checkpoint is its only route to everything.
It has to be able to hold an action for a named person, with a time limit, in a way the agent cannot release. The whole point is that a human sees the rare request, so a plain yes or no is not enough.
And it has to remember. The payment rule depends on the state of the supplier record; the delete rule depends on the state of the flag. A rule is only as strong as the checkpoint's knowledge of every action its decision depends on, and a checker with no memory cannot enforce any of this.
What it will not do
It only protects what it covers. The flag rule protects only resources that have the flag switched on, and a bank detail changed by a person logging in directly, outside the checkpoint, gets past the payment rule the same way. We reproduced that failure too, and it is the strongest argument we know for making the checkpoint the only path, not one path among several.
It does not judge quality. A correctly formed rollback to a broken release is within the rules, and it will run.
And there is at least one place where we expect the pattern to break completely: outbound communication. When the harm is what a message says, any single message can be the harmful one, and there is no rare upstream change to gate. That needs different tools, and we have written separately about one tool it cannot be.
If you know another domain where the expensive failure has no enabling change in front of it, or a reason our test systems mislead us, we would genuinely like to hear it. Argue with it here, or by email.