The real cost of cloud GPU at scale

Cloud GPU pricing looks simple on the surface: an hourly rate, a card on file, capacity on tap. For short bursts of training or experimentation, that simplicity is the whole point. The problem is that very few AI workloads stay short. Once a workload becomes sustained, the hourly rate that looked attractive in a proof of concept starts to dominate the budget conversation.

The break-even point is where this becomes visible. For most organisations running GPU-intensive workloads more than 40 to 50 percent of the time, the economics of on-demand cloud capacity begin to lose to owned or co-located infrastructure, even after accounting for power, cooling, networking and management overhead. Below that utilisation threshold, cloud remains the right call. Above it, every additional hour on a public cloud meter is effectively a premium paid for flexibility the business may no longer need.

The challenge is that most organisations do not know which side of that line they are on, because GPU spend is rarely tracked at the granularity needed to find out. Bills arrive as a single infrastructure line item, not broken down by model, team, project or tenant. Without that visibility, the question ‘should we own this or rent it’ cannot be answered with confidence, and decisions default to whatever was easiest twelve months ago.

There is also a second cost that rarely makes it into the spreadsheet: the cost of underutilisation on the other side of the equation. Owned GPU capacity that sits idle outside peak training windows is just as expensive as overpaying for cloud, just less visible because the invoice does not change month to month.

The practical path forward is rarely a binary choice between cloud and owned infrastructure. It is a structured assessment: establish real utilisation patterns over a meaningful period, model the total cost of ownership for dedicated capacity against current and projected cloud spend, and design a hybrid approach that uses owned infrastructure for baseline, predictable workloads and cloud for burst and overflow.

What this requires in practice is infrastructure that is genuinely portable between the two: consistent networking architecture, compatible storage tiers, and orchestration that does not need to be rebuilt depending on where the workload runs. Get that right, and the decision of where a given job executes becomes a cost optimisation choice rather than an architectural rebuild.

For organisations approaching or past the break-even point, the cost of inaction is not neutral. It is the gap between current spend and what the same workloads would cost under a properly designed hybrid model, multiplied by every month the decision is deferred.

There is a further wrinkle that rarely gets discussed outside specialist financial circles, but which matters for anyone trying to understand the true cost of GPU infrastructure: depreciation. The major hyperscalers do not report depreciation as a standalone line on their income statements; it gets buried across cost of goods, R&D and general administration, with no consistency between providers. That matters because the useful life assumed for a GPU has a direct and material effect on reported cost, and by extension on the pricing of the cloud capacity built on top of it. An organisation comparing the all-in cost of owning GPU infrastructure against renting it from a provider whose own cost base depends on assumptions it does not disclose is not comparing like with like. The practical implication is to build cost models on realistic, conservative useful-life assumptions for owned hardware, typically three to five years for current-generation GPUs given the pace of silicon refresh, rather than the longer depreciation schedules sometimes used for general-purpose compute.