The AI cost paradox

McKinsey senior partners Tanguy Catlin and Lari Hämäläinen highlighted a central paradox on Tuesday: intelligence is getting radically cheaper, but enterprise AI bills keep climbing anyway.

Model prices have collapsed

GPT-4 launched in two thousand twenty-three with the eight K context API priced at sixty dollars per million output tokens. Today, models performing at roughly the same level cost a fraction of that.

But usage is exploding

As the cost of producing a unit of intelligence collapses, the amount businesses consume is exploding. Models are getting cheaper per unit while enterprises ask them to perform vastly more reasoning and work, especially through autonomous agents.

Agents generate far more work

In software development, AI agents can repeatedly inspect, modify and rewrite entire codebases, generating far more code than a human developer would typically touch, Hämäläinen said.

Agent costs are wildly unpredictable

Unlike traditional software, agents can take different paths to the same result, making costs highly variable. The same task can cost up to thirty times more from one run to another, Hämäläinen said.

“The same task can cost up to 30 times more from one run to another”

Measure agents by task, not tokens

Rather than measuring agents by cost per token, leaders should evaluate them at the task level: how much a task costs to execute, how often the agent succeeds, and how much human time is required to verify its work.

When agents create value

An agent can make sense when the time required to verify its output is a small fraction of the time it would take a human to complete the task from scratch. If a task takes a human an hour but verification takes six minutes, an agent with a success rate above ten percent could create value.

Three levers for cost control

Catlin identified three areas where companies can manage AI spending: visibility into which use cases and units drive spend; optimizing workflows by matching model complexity to tasks and caching reusable context; and greater sourcing discipline including removing unused licenses and negotiating provider terms.

Optimize toward returns, not cuts

Catlin cautioned that the goal should not be indiscriminate cost-cutting. Companies should determine where AI spending delivers the highest returns and optimize toward those returns.