FinOps for AI Workloads: Controlling the Cost of Inference
Read Time 13 mins | Written by: Vinayak Bhagat
The AI line item has a habit: it starts as a rounding error, earns its keep, and then one quarter it is suddenly a number the CFO circles. Nothing went wrong. That is the uncomfortable part. The feature worked, people used it more, and every use carried a per-call price. The bill grew because the product succeeded.
This is why FinOps for AI workloads is not the same discipline as squeezing your cloud bill. Classic cloud waste is capacity-shaped: idle instances, oversized databases, storage nobody reads — things you can turn off without anyone noticing. AI spend is usage-shaped: nearly every dollar maps to somebody getting an answer. You cannot turn it off without turning the feature off. What you can do is change what each answer costs.
The goal, then, is not a smaller bill. It is a defensible cost per outcome — a number you can put next to the value of the outcome and be comfortable with as both scale. Four dials control that number. Here they are, in the order we tune them.
How do you control the cost of AI inference? Tune four dials: Model Fit — match model size to task difficulty instead of defaulting every call to the largest model; Architecture — cache repeated answers, batch what can wait, and route easy requests to cheap paths; Usage Shape — trim prompts and context, cap retries, and bound agent loops, which multiply calls invisibly; and Accountability — meter spend per feature and team from day one and manage a cost-per-outcome metric, not the raw bill. Sequence matters less than starting with metering: you cannot tune what you cannot see.
Why AI Bills Behave Differently
Three properties make AI spend harder to manage than the cloud spend your team already knows how to handle. First, it is priced per use: model APIs bill by what flows through them, so cost scales with adoption rather than with provisioning decisions. Success and spend are the same curve. A finance process built around fixed monthly infrastructure will read that curve as a problem when it is often the business working.
Second, it is shaped by engineering choices that never used to be cost decisions. How long a prompt is, how much context gets stuffed into each request, whether a failed call retries, whether an agent takes three steps or thirty — each is a product decision made far from the finance team, and each moves the bill. The pattern we see in mid-market teams: nobody is being careless, but nobody owns the connection between those choices and the invoice.
Third, a growing share of it is invisible to cloud tooling. Model APIs and AI-enabled SaaS land on credit cards and separate invoices, outside the cloud cost dashboards your team watches — and sometimes outside sanctioned procurement entirely, which is the spend problem our shadow AI governance guide exists to solve. If your FinOps practice covers general cloud waste, keep running it — our guide to cutting cloud costs without slowing engineering is that playbook. This post covers the workloads that playbook was not built for.
The Four Dials of AI Spend
Each dial is owned by a different role, which is exactly why AI cost work fails when it is assigned to one team. Finance cannot resize a prompt; engineering cannot decide what a resolved support ticket is worth. The dials give each owner a concrete surface, and the unit metric at the end ties them together.
Dial 1 — Model Fit: match the model to the task
The most expensive default in AI engineering is routing every request to the most capable model available. Frontier models earn their price on genuinely hard tasks — multi-step reasoning, nuanced drafting, ambiguous inputs. A large share of production traffic is not that: classification, extraction, reformatting, routine summaries. Smaller and mid-tier models handle those at a fraction of the unit cost, and the gap between model tiers is one of the largest cost levers that exists in this space.
The check: for each AI feature, can the team show an evaluation — even a lightweight one — that justified the model choice against a cheaper tier? "We used the big one to be safe" is a research decision that became a production bill. The fix is an evaluation habit: define what good output looks like, test the cheaper tier against it, and downgrade wherever quality holds.
Dial 2 — Architecture: stop paying for the same answer twice
Three patterns do most of the architectural work. Caching: when many users ask materially the same question, compute the answer once — response caches and prompt caching both exist for exactly this. Batching: work that does not need a real-time answer (nightly enrichment, bulk classification) can run through batch paths that providers price lower than interactive calls. Routing: put a cheap step in front that sends easy requests to a small model or a non-AI path entirely, and reserves the expensive model for requests that need it.
The check: ask the team what percentage of requests are served without touching the most expensive model. If the answer is "we have never measured that," this dial has not been turned. None of these patterns degrade the user experience; they are invisible everywhere except the invoice.
Dial 3 — Usage Shape: the quiet multipliers
Because model usage is priced by volume, everything that inflates a request inflates its cost: system prompts that grew for a year without an edit, entire documents attached where a relevant excerpt would do, conversation history replayed in full on every turn, silent retries that bill twice for one answer. Each is small. Multiplied across every call, they become the difference between a viable unit cost and an inexplicable one.
Agentic systems raise the stakes: an agent that plans, calls tools, checks its work and re-plans makes many model calls per user request, so a loop that wanders is a bill that wanders. Uncapped steps and unbounded retries are precisely the kind of thing our production-readiness checks for agentic AI gate before go-live. The check: every agent has a step budget and a cost ceiling per task, and someone can state both numbers from memory.
Dial 4 — Accountability: meter it like a product cost, not an IT cost
The first three dials are worthless if nobody can see them move. Accountability means every AI call is tagged to a feature and a team from day one; spend rolls up into a view finance and engineering both look at; each feature carries a unit metric — cost per resolved ticket, per generated draft, per qualified lead — and budgets alert before quarter-end instead of explaining afterwards. This is standard FinOps discipline — the FinOps Foundation frames it as making cost a first-class engineering metric — applied to a spend category young enough that most companies have not wired it in yet.
The check: can you answer "what did feature X cost last month, and what did each outcome cost?" in one query? If AI spend arrives as one undifferentiated line, every future cost conversation will be an archaeology project. Tagging is an afternoon of work when a feature launches and a quarter of work retroactively.
| Dial | What turns it | Primary owner | The check |
|---|---|---|---|
| 1 · Model Fit | Evaluations that right-size the model per task | Engineering / ML lead | Each feature's model choice is justified against a cheaper tier |
| 2 · Architecture | Caching, batching, routing in front of expensive calls | Engineering | You can state what share of requests skip the top-tier model |
| 3 · Usage Shape | Prompt and context discipline; retry and agent-loop caps | Product + engineering | Every agent has a step budget and a per-task cost ceiling |
| 4 · Accountability | Per-feature metering, unit metrics, budgets and alerts | Finance + engineering (FinOps) | "What did each outcome cost last month?" is one query away |
The Three Mistakes That Inflate AI Bills
Mistake 1: Waiting for the bill to hurt before metering. Tagging and unit metrics feel like bureaucracy while the spend is small — which is exactly when they cost nothing to add. Teams that meter from the first feature never have the "what is this number" meeting; teams that wait reconstruct months of untagged spend during it.
Mistake 2: Optimizing the model before measuring the loop. Swapping to a cheaper model while an agent silently retries and re-plans is tuning the smallest dial first. Instrument calls per request before touching model choice; in agentic systems, the loop is frequently the bigger number.
Mistake 3: Making cost control so heavy that teams route around it. If every experiment needs a procurement cycle, builders reach for personal accounts and free tiers, and the spend leaves your field of view entirely — the shadow AI problem, now with an invoice. Give teams a sanctioned sandbox with a budget and visibility instead of a wall.
Where to Start Monday
Start with Dial 4, even though it saves nothing by itself: tag every AI call by feature, pick one unit metric per feature, and set an alert threshold. Visibility first — you cannot tune what you cannot see. Then instrument calls-per-request on your busiest feature and cap anything unbounded (Dial 3). Then measure what share of traffic actually needs the expensive model and put a route or cache in front of it (Dial 2). Model fit (Dial 1) comes last, once evaluations exist to protect quality while you downshift.
Run in that order, the work compounds: each dial makes the next one's effect measurable, and the output is a cost-per-outcome number that survives scrutiny in a board meeting. That is the standard our FinOps practice holds engagements to, and the same discipline our generative AI team builds into systems before they ship rather than after the first surprising invoice.
Get ahead of the AI line item
Ontrac's FinOps team runs the four-dial review on your AI workloads: where the spend actually goes, which dials are untouched, and a sequenced plan to a defensible cost per outcome — without slowing the teams shipping the features.
Talk to the FinOps teamFrequently Asked Questions
What is FinOps for AI workloads?
The practice of making AI spend visible, attributable and tunable: metering model usage per feature and team, managing a cost-per-outcome metric rather than the raw bill, and treating prompts, context, retries and model choice as the cost levers they are. It extends standard FinOps discipline to a spend category that is priced per use and shaped by engineering decisions.
Is calling an AI API cheaper than hosting our own model?
At mid-market volumes, usually yes — APIs carry no infrastructure, no ML-ops staffing, and no idle capacity, and you pay only for what flows through. Self-hosting starts to justify itself with very high steady volume, strict data-residency requirements, or latency needs APIs cannot meet. Run the comparison on total cost including people, not on a per-call price alone.
What unit metric should we track for AI spend?
Cost per business outcome, defined per feature: per resolved ticket for a support assistant, per accepted draft for a writing tool, per qualified lead for enrichment. Cost per request is a useful engineering diagnostic but a poor management metric — requests are not what the business buys. The right unit metric makes "the bill went up" answerable with "and here is what we got for it."
Who should own AI cost management?
A pairing, not a person: finance owns the budgets, unit economics and review cadence; engineering owns the dials that move the numbers. If your organization already runs a FinOps practice, AI workloads join it as a new cost category with new levers. If it does not, the AI bill is usually the forcing function that starts one.
References
FinOps Foundation — What is FinOps: finops.org
This article describes general patterns from Ontrac Solutions' consulting work with mid-market organizations. It contains no client-specific data and cites no per-token prices, which change frequently; evaluate current provider pricing against your own workload profile before making architecture or contract decisions.
