What the engine stopped, and what it missed

Question

An agent that holds a company’s credentials will do what it is told, including by the wrong person. The usual answer is to make a person approve everything, which nobody keeps up for long. We wanted to know whether a layer that checks each request against the agent’s mandate, and asks a person only when something is off, would stop most wrongful payments without burying people in approvals. And we wanted to know, just as much, what it would let through.

Method

The population

300 firms, 144,517 people in all: 141 in the UK, 89 in the EU and 70 in the US, across ten sectors from logistics to healthcare. Each firm ran 2 to 30 agents, more in larger firms, and each agent was given one mandate built from the console’s own templates (Table 1).

The engine

Every request went through the decision code the gateway runs: the same mandate checks, risk signals and autonomy tiers as production, 744,877 decisions in all. Every batch of model calls, 92 million of them, went through the same router and budget ladder. What the simulation does not include is the hosted service around that code: there is no network, no database and no Slack, and the people answering approvals are modelled rather than observed.

The comparison

Each request was run twice: once with no gate, where the agent does what it is asked, and once through Immiscible. Existing call-back checks on changed bank details catch some fraud in both worlds. Payments in euros and dollars are counted in pounds at the engine’s reference rates.

The attacks

Business email compromise follows the AFP payments fraud survey¹ and the FBI’s complaint data²; duplicate invoices, APQC’s benchmark for payments that already get past a firm’s controls³; address poisoning, the USENIX study of how poisoned addresses are built⁴; prompt injection, the rate at which a capable agent obeyed injected text in AgentDojo⁵; and unmanaged keys, Microsoft’s finding that most people bring their own AI tools to work⁶. Where no study gives a rate, we chose one and wrote it down (Appendix A).

Results

Money

Without a gate, £15.9m would have left the 300 firms in 6,073 wrongful attempts. Through Immiscible, £13.4m stayed put: 3,022 attempts refused by the engine outright, and 2,202 held and then refused by a person. Prompt injection was the largest kind by value. Crypto address poisoning was stopped most completely, 96% of its value, and duplicate invoices least, at 69% (Figure 1, Table 2).

Data and secrets

All 566 attempts to send customer lists to an outside address, 11,637,504 records in all, were refused, because no CRM rule named the address. Coding agents tried to send secrets out 968 times; 876 were refused and 92 went to a host the coding rule allows, so they got through.

People

Most of the work needed nobody (Figure 2). When a person was asked, it was most often about a payment above what the agent may pay alone, and they answered in minutes during the working day, or the next morning otherwise. Across the run, the gate asked for 28% of the approvals a policy of approving every payment and data release would have (Table 3). The cost fell on legitimate work too: 22.1% of legitimate actions waited for a person, and 1.0% were refused.

Table 3 counts more approvals (140,321) than Figure 2 counts decisions held for a person (135,924). The difference is model spend: 4,698 times a person was asked whether a team could go on spending once its budget ran out, or whether a runaway loop could carry on. Most of those were budget checks rather than decisions on an action, and Figure 2 counts only decisions.

Model spend

The same agents spent £4.4m on models without the gateway and £1.7m with it, 61% less (Table 4). Routing each task to the cheapest model that clears its capability floor did most of that; budgets, loop limits and finding unmanaged keys did the rest. Of all the numbers here this is the least certain. It depends almost entirely on which models a firm starts on, and we did not model the quality of output above each task’s floor.

What it missed

£2.5m got through, in 757 attempts. We think this is the most useful part of the note, so we have tried to be exact about it.

The largest share, £1.4m in 503 attempts, was duplicate invoices. The engine holds a payment of the same amount to the same payee within 72 hours, and a person then decides. In the model, 40% of duplicates are resent later than that, between 4 and 30 days, and nothing in the engine connects them to the first payment. A longer window would catch more of them and hold more legitimate repeat payments; we have not yet measured that trade.

Prompt injection let through £516k in 212 attempts, changed bank details £421k in 15, address poisoning £45k and lookalike domains £90k. Each got through in one of two ways. Either the engine held the payment and a person approved it without seeing the fraud, or the engine allowed something that, by every signal it has, looked legitimate: a bill planted among the ones the agent reads as approved is, to the engine, a real bill. Table 5 in Appendix B shows individual cases of both.

Outside payments, the 92 secret uploads that reached an allowed host are a miss of a different kind: the rule was too broad, and the engine did what the rule said.

Limitations

This is one run with one seed. Most of the rates are our assumptions, and real attack mixes vary widely from firm to firm. People are modelled as probabilities, so tired approvers, fraud that adapts to the gate and insiders are absent. A loss stopped here might also have been recovered later in the real world, and a legitimate payment refused here would usually be paid a day later. Above all, these are not customers. We will publish customer results when we have them, with their permission, and expect them to differ.

Reproduce

The simulation ships in the repository and needs nothing but Node. Running it prints the same figures and the same decision digest; test/society.test.js fails if the engine ever decides differently from what this note reports.

Command

node src/sim/society.mjs

Seed

20261006

Decision digest

2c48dd25e7eaa3e914b644aca0802debe97047fac486876447502834b2fae5cf