Paying suppliers, with an agent doing the work

The scenario

A mid-sized company receives supplier invoices by email. Someone used to open each one, check it against the purchase order and the delivery note, and release the payment. That work is repetitive and well-defined, so it is an obvious thing to hand to an AI agent: read the mailbox, find the order, compare the amounts, pay.

The agent does this well. It can also be steered by whatever it reads, which is the problem, because one of the things it reads is email from outside the company.

How it goes wrong

It goes wrong in two ways, and only one of them involves an attacker.

The mistake. A duplicate invoice, a wrong currency, a decimal point in the wrong place, an invoice paid before the goods arrived. Nobody is attacking; the agent is working from an incomplete picture, at machine speed, across hundreds of documents.

The redirection. An email arrives that looks like it comes from a real supplier: we have changed bank, please update our details. Sometimes it carries a plausible letterhead; sometimes the sender's mailbox has actually been compromised, so it comes from the right address. The bank details are changed. Then the next genuine invoice from that supplier arrives, is matched correctly against a real order for a real delivery, and is paid — to the attacker.

This is the shape that is easy to miss: the fraudulent step is not the payment. The payment is correct in everything that can be checked on the payment itself: real supplier, real invoice, real order, right amount, within limits. The fraud happened earlier, in a small administrative change that no payment rule was looking at.

Where the checkpoint goes

The agent stops talking to the finance system directly, and it stops holding the credentials. Every request it makes — read this invoice, look up this order, change this supplier's bank account, pay this invoice — goes to the checkpoint first. The checkpoint tests the request against rules your own people wrote and approved, then carries it out or refuses it, and records the attempt either way. Nothing about the agent changes: same model, same instructions, same tools by name.

Seen from the checkpoint, a normal week looks like this:

The rule that stops it is five lines long

The rule that stops the redirection is not about payments at all. In plain terms: any change to a supplier's bank account waits for the finance controller, and if nobody approves it within a day, it does not happen.

Supplier.updateBankAccount:
  requireApproval:
    approvers: role:finance-controller
    timeout: 24h
    onTimeout: deny

This works because of an asymmetry. A supplier's bank details change perhaps a handful of times a year, and when they do, somebody should look. The payments themselves — hundreds a month, most of them small and ordinary — stay automatic. Guarding the payments would mean reviewing hundreds of items a month and still missing this fraud, because each payment is individually correct. Guarding the bank-detail change means reviewing a handful of requests a year, and the fraud cannot happen without one of them crossing the controller's desk.

We have seen the same shape in every domain we have modeled: the control belongs on the small, rare change that unlocks the large, frequent effect, not on the effect itself.

Alongside it, ordinary rules do the ordinary work: a ceiling on payment size with a matching daily total, so money cannot be split into smaller payments to slip under the limit; a requirement that each invoice match one open order line, which the payment then uses up, so the same line cannot pay two invoices; and a refusal to pay countries the company does not trade with. Together they close the flanking move as well: inventing a new supplier does not help the attacker, because a fresh supplier has no open order line for a payment to match.

What you get

• The fraud cannot happen quietly. The agent can be completely taken in by the email. It can ask for the change and argue that it is reasonable. The change still waits for the controller, because the agent no longer has the ability to make it alone.
• Every attempt leaves a record, including the refused ones. What was tried, when, on whose behalf, and which rule stopped it. Without that record, a blocked attack looks the same as a week in which nothing happened.
• Your reviewers approve rules, not a model. "Do we trust the AI?" is not a question anyone can answer. "Do we accept these rules?" is ordinary sign-off work your organization already knows how to do.
• One switch stops everything. If the agent starts behaving oddly at 2am, an operator halts it with a single action, and anything already queued to run stops with it.

What it will not do

• It cannot tell you the invoice itself is wrong. A duplicate of last month's invoice, or a bill for goods nobody received, still gets paid if it matches an open order. That payment is within the rules, so it goes through.
• It only checks what passes through it. If bank details can also be changed by a person logging into the finance system directly — often they can — that change is never checked, and the payments after it look correct here too. Making the checkpoint the only path for these actions is part of the rollout, and it is work in your systems, not a feature anyone can ship to you.
• Some of the value is already on your shelf. Several of the losses we modeled disappear by switching on controls the finance system already has. Do that first. What the checkpoint adds is the case where a person's job was to notice something odd, and no person was ever going to see it.