Traditional cloud FinOps grew up around a relatively predictable cost model: compute, storage and network scaled roughly with usage, and reserved capacity or committed-use discounts smoothed the rest. Generative AI breaks that model. Inference cost varies with prompt length, model choice, retrieval overhead and user behaviour in ways that are far harder to forecast, and the unit economics can swing by an order of magnitude between a well-tuned system and a careless one. Organisations that treat AI spend as just another cloud line item are routinely surprised by the bill, usually after the workload has already gone live.
Why AI workloads break existing cost models
A conventional application's cost is dominated by infrastructure that runs continuously regardless of load. An AI application's cost is dominated by the marginal cost of each request, which depends on model size, context window, output length and whether the request hits a cache or a cold model. Two features that look functionally identical to a user, such as a short FAQ answer and a long document summary, can differ enormously in cost, and that variance is invisible in most existing dashboards because they report infrastructure spend rather than cost per outcome.
GPU capacity compounds the problem. Reserved GPU instances carry a steep cost whether or not they are fully utilised, on-demand GPU capacity is expensive and sometimes scarce, and the gap between the two creates a genuine planning problem rather than a simple purchasing decision. Leaders who do not have visibility into utilisation end up either overpaying for idle reserved capacity or absorbing volatile on-demand pricing at the worst possible moments, typically when a product launch drives a spike in usage.
Building unit economics that mean something
The starting point is to stop measuring AI spend in aggregate and start measuring cost per meaningful unit of work, such as cost per resolved support ticket, cost per document processed or cost per successful task completion. Aggregate spend tells leadership almost nothing about whether the system is becoming more or less efficient; unit economics tell them immediately whether a model change, a prompt change or a traffic pattern has made the product cheaper or more expensive to run.
This requires instrumenting the request path to capture token counts, model version, cache hit rate and latency alongside cost, and joining that data to product usage so that finance and engineering are working from the same numbers. Without this join, finance sees a cloud bill and engineering sees a model performance dashboard, and neither can explain the other's view of the system.
Where the savings actually come from
In most production systems, four levers account for the majority of achievable savings, and they are worth pursuing roughly in this order because of how quickly they pay back. Prompt and context engineering that trims unnecessary tokens from every request compounds across millions of calls. Aggressive caching of repeated or near-duplicate queries removes a meaningful share of inference calls entirely in high-traffic applications. Model routing, which sends simple requests to smaller, cheaper models and reserves the largest model for genuinely hard tasks, can cut blended cost substantially without a noticeable quality drop for most users. Batching and request scheduling for non-interactive workloads improves GPU utilisation and reduces the premium paid for on-demand capacity during peak periods.
None of these levers is a one-off project. Each needs an owner, a metric and a review cadence, because model providers change pricing, usage patterns drift and yesterday's optimal routing configuration quietly becomes suboptimal as the product evolves.
Capacity strategy: reserved, on-demand and provider mix
Capacity planning for AI workloads benefits from the same discipline applied to any volatile, high-value resource. Establish a baseline of reserved or committed capacity sized to sustained average load, use on-demand or spot capacity to absorb predictable peaks, and maintain the ability to route across more than one model provider so that a price change or capacity constraint at one provider does not become a single point of failure for cost or availability. This multi-provider posture also creates negotiating leverage that a single-vendor commitment gives away.
Forecasting should be revisited far more often than a traditional infrastructure budget, because usage of a successful AI feature can grow non-linearly once it embeds itself in a workflow. Treat the first two or three months after launch as a calibration period, with tighter budget reviews than a mature workload would warrant.
Governance: making cost a design constraint, not an afterthought
The organisations that keep AI spend under control build cost review into the same gate as security and quality review before a feature ships, rather than discovering the economics after launch. Every new AI feature should have an estimated cost per unit of value and a threshold at which it gets flagged for optimisation or reconsideration. Product and engineering leaders should see cost alongside latency and accuracy in the same dashboard, so that a decision to use a larger model is made with the cost trade-off visible, not buried in a separate finance report that surfaces weeks later.
- Instrument every AI request with token counts, model version, cache outcome and cost, joined to product usage data.
- Report cost per meaningful unit of work, not aggregate spend, to product and engineering leadership.
- Apply prompt trimming, caching and model routing before adding capacity to solve a cost problem.
- Blend reserved and on-demand GPU capacity, and maintain the ability to route across more than one model provider.
- Recalibrate forecasts frequently in the first months after launch, when usage can grow non-linearly.
- Put a cost-per-outcome estimate in front of every AI feature decision, alongside latency and accuracy.
Common pitfalls
The most common mistake is treating AI infrastructure spend as a scaled-up version of familiar cloud costs, which leads teams to apply reserved-instance thinking to a workload whose marginal cost dominates. A close second is optimising model choice for benchmark accuracy alone, without weighing the cost difference against the marginal quality gain for the specific use case in production. Many teams also delay cost instrumentation until the bill becomes a problem, by which point the workload has scaled and habits are harder to change. The fix in every case is the same: treat cost as a first-class metric from the day a generative AI feature is designed, not a line item reconciled after the fact.
Need support applying this approach? Email sales@halfteck.com.