essay
A familiar procedure, applied to a new kind of worker
Companies have always handed real work to people who make mistakes. The machinery that makes that safe is centuries old, and AI agents are being deployed without it.
People making mistakes is an old, well-managed problem. Long before software, companies learned to run on workers who are sometimes careless, sometimes confused, and occasionally dishonest. The tools they built for it are older than the modern company: the ledger, the second signature, the spending limit, the audit.
Those tools were never an expression of distrust in any particular clerk. They are what made it possible to hand a clerk real money in the first place. Nobody proves the clerk incapable of error before handing over the cash box; the procedure limits the damage an error can do, and once the damage is limited, the company can afford to delegate the work.
The control never asks why
The ways this goes wrong are familiar. A careless clerk pays an invoice twice. A confused clerk fills a gap in the paperwork with a guess. A dishonest clerk, or one taken in by a fraudster, moves money where it should not go. The company survives all three, and it never has to work out which one it is dealing with: the spending limit holds whether the clerk was tired, mistaken, or lying.
That is the property worth carrying over, because an AI agent fails in the same three ways. It makes mistakes, at machine speed. It fills missing information with plausible invention. And it can be talked into things by the text it reads, which is a problem when some of that text comes from strangers. Arguments about which failure a given model is prone to, and how often, do not have to be settled first. A control that simply assumes mistakes will happen covers the model you run today and the different model you will run next year.
The machinery did not transfer
Agents are often deployed without any of this. An agent can be wired into live systems with privileges a trained and trusted employee would never be given: no second signature, no spending limit, no ledger anyone reads. That is usually not a decision anyone made. The technology made the wiring easy, and nothing in it asks for the procedure: hiring a clerk takes a contract and onboarding, but an agent can be given the keys to production systems in an afternoon.
Most of the old machinery carries over almost without translation. A spending limit becomes a maximum amount checked by software. Approval by a second person becomes an action held until a named person releases it. The rule that no one person controls a whole payment — finance calls it segregation of duties — becomes a policy where the agent that pays invoices cannot also change a supplier's bank details unreviewed. The ledger becomes an audit record written on every attempt, including the refused ones; that is one place where software improves on the paper original, because a paper ledger records what happened, and this one also records what was tried.
One thing does not transfer: the social pressure. A human clerk works inside raised eyebrows and the knowledge that the auditor comes in March. An agent feels none of that, works in parallel, and reads its instructions from whatever lands in the mailbox. So the procedure cannot be advisory. It has to sit in the path of the action itself, as software, where no text the agent reads can argue with it.
What it will not do
It limits damage; it does not do the work, and it does not judge the work. A payment that matches its order goes through even if the goods arrived broken, exactly as it would under a clerk, and for the same reason: the records said it was owed. And the procedure only covers what goes through it. A clerk with a second bank card nobody knows about is outside the company's control, and an agent holding its own credentials is the same clerk.
None of this is a new theory of control. It is the oldest one there is, applied to a new kind of worker, and its age is the argument for it: these controls have survived generations of unreliable workers, and they do not assume the next model will be better than this one.
The worked versions of this argument — supplier payments, cloud operations, work queues — are in the use cases, each with what the control buys and what it does not. The companion essays: An LLM reviewing an LLM is not a safety bound, on why the popular alternative does not give you the same property, and Gate the enabling change, on where the rules belong.