One Big Idea

Your AI got cheaper. Much cheaper. And your bill did not go down. That is not a billing error, and — this is the part most banks get wrong — it is not a cost problem either. It is a forecasting problem, and a forecasting problem is worse, because a cost problem can be budgeted.

Last week this letter asked why a control that passes is not the same thing as a bank that is protected. This week the question moves from what a control can see to what the work costs — and specifically to whether the bill can be predicted at all. The honest answer, for most banks, is not yet.

A cost problem can be budgeted: you can decide to spend more. A forecasting problem cannot, because there is no number to put in the budget.

The Insight · It is the loop, not the model

Most banks fail here for one reason: they treat it as a cost problem and go negotiate a better rate. That instinct is wrong in a specific way — the rate was never the issue.

Frontier model prices really have collapsed. On the BenchLM Token Price Index, the frontier sub-index sits roughly eighty percent below its March 2023 base (data from BenchLM.ai). Read that precisely, because the precision is the point: it is an index reading, not any model's price, the March 2023 base applies to that series alone, and no model is named at a price anywhere in this issue. And over the past year the index has started climbing again, as each new generation of capability arrives at a higher price than the one it replaced. The collapse already happened. It did not fix the bill.

So where does the money go? A price per token describes an input. A bank buys outputs: a reconciliation finished, a case resolved, a document checked, an exception cleared.

A chatbot receives a question and writes an answer. An agent plans, calls a tool, reads the result, checks its own work, retries — and on every turn of that loop it re-reads everything it has already done. The context is resent, and resent, and resent.

The measurements are striking and their denominators matter. Researchers including Erik Brynjolfsson and Alex Pentland measured agentic workloads consuming roughly one thousand times more tokens than an ordinary coding-assistant exchange, with input tokens rather than output driving the cost — Bai et al., arXiv:2604.22750, measured on agentic coding tasks across eight frontier models. A second study found fifty-nine point four percent of tokens going to the iterative review stage, the machine checking its own output rather than producing it; that one is a preliminary finding, averaged over thirty software-development tasks in a single agent framework on a single model.

Read the mechanism as general and the magnitudes as the documented shape of the problem, not as your bank's numbers. Bai et al. also report that higher token usage does not buy higher accuracy, which removes the comfortable reading that the spend is simply buying quality.

Gartner named this in August 2026 and calls it the Inference Paradox: better unit economics subsidise more complex workflows, so total cost climbs even as each unit gets cheaper. Their forecast is inference cost per agentic workflow rising more than fivefold through 2028.

Framework of the Week · The AI Unit-Economics Ledger

This is the author's proposed decomposition of where AI cost actually accrues — not a validated standard, not four products to buy, and not a maturity ladder. Four lines, not six, because a chief financial officer can hold four in their head. They are ordered by when a bank finds out about them, which is what makes the ledger diagnostic rather than descriptive:

  • L1 · Inference — the model call itself. The only line most approved cases contain, and the line whose unit price has fallen furthest, which is exactly why it flatters the forecast. Found at approval.
  • L2 · Retrieval and context — embedding, vector storage, re-ranking, and the context resent on every call. It scales with your corpus, not your user count, so it appears nowhere in a per-user business case. Found at integration.
  • L3 · Orchestration — the multiplier: planning, tool calls, retries, self-checks, handoffs. Everything in the section above lands here. Found in production, which is the worst place to find it.
  • L4 · Run, assure and review — observability and drift monitoring; the judge models and regression suites re-run on every release; and the human part — exception handling, escalation, disputes, appeals. Found at audit.

Review is named deliberately, and it is the limb most often deleted. A line called "run and assure" reads as infrastructure and gets delegated to engineering. A line called "run, assure and review" cannot be — because review is where the headcount saving your business case promised either survives or quietly disappears.

One absence worth knowing about. There is a standard for AI cost data — the FinOps Open Cost and Usage Specification. It standardises billing and usage schemas, which makes the numerator comparable whoever billed it. It defines no cost-per-task metric, and never claimed to. The industry standardised the numerator. The denominator is yours to build.

Use Case · The same task, thirty times apart

The obvious objection is that this is a modelling problem — that a good enough estimate, with a prudent buffer, closes the gap. So look at what happens when researchers run the same work twice.

In the token-consumption study — Bai and colleagues again — the researchers ran the identical task more than once. Not variations of it, not a harder version: the same task, repeated. Token consumption varied by up to thirty times.

That finding should change how a committee reads a number. The invoice for a piece of work is not a figure. It is a distribution. The same reconciliation, run on Tuesday and again on Thursday, can differ by a factor of thirty, and nothing in your change log will explain why.

And you cannot simply ask the system. The same researchers tested whether frontier models can predict their own token usage. The correlations were weak to moderate — up to zero point three nine — and the models systematically underestimated. The thing generating the cost does not know what the cost will be, and errs cheap.

One caution on scope, since this letter keeps asking you to check denominators. The strongest measurements available — the thousandfold multiplier, the review concentration, the thirty-times variance — were taken on agentic coding tasks, not on credit files or reconciliations. The mechanism generalises; the specific magnitudes should not be assumed to transfer unchanged to a banking workload. That is an argument for measuring your own, not for dismissing these.

Risk Note · Did the headcount saving arrive?

This is the question a chief financial officer actually asks, and it is where the evidence is thinnest and loudest at the same time.

The widely-cited cases — the ones about replacing hundreds of service agents — trace back to the deploying companies' own announcements and to their technology suppliers, presented as forward-looking projections rather than audited results. Where those projections have later been revisited, the revision has run downward, and none has been restated as an audited reduction in a regulated process's cost base. The public record does not support a sustained net reduction in the cost base of a regulated banking process through headcount substitution.

The mechanism explains why, and it is more useful than any single case. Automate the straightforward path and what remains is the hard residue — the disputes, the exceptions, the appeals. That work needs more expensive people than the triage it replaced. The cost did not vanish. It moved into line four, where nobody forecast it.

Two honest limits on the argument. A bank with genuine engineering maturity — disciplined inference tiering, cache-aware agent architecture, model routing — can make falling per-unit prices outrun orchestration bloat. The controls are real, and routing is Gartner's own recommended remedy, which makes it a control rather than a refutation. But very few traditional commercial banks run those architectures in production today, so the binding constraint may be talent, not price — and that is the strongest argument for the ledger rather than against it, because a bank that cannot yet engineer the optimisation must at minimum measure the exposure.

Take one approved AI business case. Any one. Look for line two, line three and line four.

When only line one is forecast, send the case back — not because the bill is too large, but because it is not a forecast. It is an estimate wearing a forecast's clothes. And ask for the variance rather than the average: a single number for a workload with thirty-times spread is the defect, not the starting point. Then give every line a named owner who answers for it.

Latest Video

This week's briefing — Video 17: Why cheaper AI cost more — walks the four lines and the four gates through on screen in 8 minutes 0 seconds, in the order a bank actually finds out about them: inference at approval, retrieval at integration, orchestration in production, and run-assure-review at audit.

Watch: youtu.be/lkfZWBZcUxI

The full deep-dive — the Inference Paradox in Gartner's own framing, every magnitude with the denominator it was measured on, the four lines with their gates, the FOCUS specification's real scope, and the headcount question read against the public record — is in The AI Unit-Economics Ledger: Why Cheaper AI Costs More.

The free five-page playbook in the Frameworks library turns it into a worksheet for your next investment committee — the four lines, the four gates, and the six-question forecast audit you can put in front of a business case before it is approved.

Reply and tell me which of the four lines your last approved AI business case actually forecast. Not which ones you could estimate afterwards — which ones were in the paper the committee signed. In most banks it is line one, alone. I read every response. Forward this to someone about to approve an AI business case on a per-call estimate.

Was this forwarded to you? Subscribe to The AI Architect Letter — free, every Saturday.

Minh Tran · AI Business Architect · LinkedIn · Workshops & advisory: aibusinessarchitect.ai