Per-model cost allocation done right

As AI workloads move from a handful of experimental projects to dozens of models running in parallel across teams, a question that used to be academic becomes urgent: who is actually spending what, and on which model?

The honest answer, in most organisations, is that nobody fully knows. GPU infrastructure tends to be provisioned at a cluster or environment level, while consumption happens at the level of individual models, experiments and inference workloads run by individual teams. The gap between those two levels of granularity is where cost visibility goes to die.

This matters for reasons beyond tidy accounting. Without per-model cost data, there is no reliable way to answer questions that increasingly drive real decisions: which models are worth their compute cost in production, which experiments are quietly consuming a disproportionate share of shared capacity, and which teams need additional budget versus which teams need a conversation about efficiency.

The instinct when this problem becomes visible is often to solve it with process: require teams to log their usage, submit cost estimates, or request capacity through a central allocation function. This tends to slow everyone down without actually producing accurate data, because the underlying infrastructure still cannot see what it is being asked to track.

The more durable approach is to build allocation into the infrastructure itself. Tagging at the point of resource allocation, whether that is a Kubernetes namespace, a scheduler queue or a tenant boundary, so that consumption data is attributable automatically rather than self-reported. Combined with a scheduler that enforces those boundaries rather than treating them as advisory, this turns cost allocation from a reporting exercise into a byproduct of how the infrastructure already operates.

The result, done well, is a model where teams retain the autonomy to run their own workloads without friction, while finance and infrastructure leadership get accurate, near-real-time visibility into where AI spend is actually going, down to the model and tenant level. That visibility is what makes the rest of AI FinOps possible: rightsizing, chargeback, budget forecasting and the underutilisation conversations covered elsewhere in this series all depend on having this data in the first place.

Getting this right early, while the number of models in production is still manageable, is considerably easier than retrofitting it once dozens of teams have built workflows around infrastructure that was never designed to be measured this way.

There is one further variable worth folding into per-model cost models, and it is one that has become harder to ignore over the past year: storage cost volatility. Flash storage pricing has moved sharply in recent quarters, driven by AI demand pulling memory manufacturing capacity away from the storage tiers that AI pipelines themselves depend on for checkpointing and dataset throughput. A cost allocation model that treats storage as a fixed, background cost will understate the true cost of data-intensive models and overstate the relative cost of compute-heavy ones. Building storage consumption into the same per-model tagging and attribution framework as GPU time, rather than allocating it separately or not at all, keeps the overall picture honest as input costs shift.