Token pricing is one of the most important pricing innovations in the AI era. It turns inference into something you can buy without managing infrastructure. It makes providers easily comparable. It unblocks thousands of teams who ship production AI but don’t want to  commit to capacity before they have traffic. Because of these reasons, a vast share of today's AI inference is priced per token.

For these reasons and more, any evaluation of AI cloud providers starts with a number: price per million tokens. However, token pricing is incomplete. The reason is that a good model that is more expensive per token may ultimately cost less than a bad model that is cheaper per token. Price per token is a description of what is being sold, not necessarily what you need.  Taking the sticker at face value doesn’t answer the bigger question: what is my inference costing me, really. 

Take a customer-facing support assistant. Two models can do the job, and one lists at half the per-token price of the other, so based on the sticker it's the obvious pick. But a token from one model isn't the same as a token from another. These models are non-deterministic, so the same question doesn't always come back usable: sometimes the answer is right, sometimes it's a confident hallucination, sometimes it takes a second call to land. Say the cheaper model returns a usable answer 70% of the time and the pricier one 90%. The cheap sticker is now generating retries, extra tokens, and answers you can't ship, while the "expensive" model resolves more on the first pass. Measure the number that matters, cost per answer your product can actually use, and the ranking flips. 

That gap between the sticker price and what you pay to generate useful output within the requirements for your specific workload is the token pricing illusionWhat is a useful token?

What is a useful token?

Before we can talk about the illusion, we need a unit of value that the accounting can actually be built on. We'll call it the useful token—a token that meets three conditions:

• It arrived inside your latency SLO. p95 or p99 time-to-first-token and inter-token latency within the bounds your product committed to users.
• It landed in the right context. The session, agent loop, or batch window it was generated for—not after a timeout.
• You paid for it once. It wasn’t duplicated by a client retry, a fallback model call, or a provider-side restart.

In short, a useful token is a token your product got value from. And it's important to remember that how cheaply you get a useful token depends on caching too. In agent loops, multi-turn chat, and RAG, the same prompt and context repeat constantly, and whether they're served from cache or re-billed at full rate can swing your real cost per useful token more than the sticker. Any honest pricing conversation starts by asking what a useful token costs you. That's where the sticker price starts to break down.

Three things the sticker price abstracts

1. SLO misses

Every token gets billed the same, whether it arrived inside your latency SLO or after the user gave up, whether it landed in the right context or timed out, whether you paid for it once or a retry billed you twice. The tokens that don’t generate value are still on the bill, so your effective cost per useful token sits above the sticker price you signed up to.  

2. Idle capacity rolled into the sticker price

Tokens are priced against the average workload, and that price needs to leave headroom for the  idle time of every tenant pooled on shared endpoints. That blend is a bargain when your utilization runs below it but it turns into a hidden cost for steady, high-volume workloads that could keep dedicated capacity busy most of the day. Your effective cost per useful token ends up higher than the same workload would cost on capacity you actually fill.

3. Autoscaling overhead

When traffic jumps, capacity has to come online: pod spin-up, model load, weight transfer from storage to GPU. That takes time, and on a per-token bill the time is invisible. It doesn't show up as a line item; it shows up as latency while new capacity warms up. How much it actually costs you comes down to how fast the infrastructure underneath can scale, which is the kind of thing the sticker can't show you either.

The metric that matches reality

If a useful token is the unit that matters, the right cost measure is obvious:

This isn't a new pricing model. It's a diagnostic. Compute it on whatever pricing structure you're on and it returns the honest number: what your product actually experienced, regardless of how the bill was formatted. Now you’re looking past the sticker price to the actual economics. Token pricing isn't the problem,  applying it to every workload is.

Where token pricing is still exactly right

For a specific set of workloads, token pricing makes perfect sense:

• Exploration and iteration: you don't know your traffic shape before you have traffic. Token pricing is the rational choice while you're working out prompts, retrieval strategies, and agent architectures. You pay for what you use; you don't lock into capacity you do not need.
• Variable or bursty traffic: When utilization would sit low or is unpredictable on any dedicated setup, token pricing is honestly priced for your reality. You're paying for optionality and in this scenario, it’s worth paying for.
• Sub-threshold volume: Below a certain volume, dedicated capacity economics simply don't beat token pricing. The diagnostic tells you exactly where the threshold is and below it, token pricing still wins.

This is why CoreWeave Inference includes a token-priced path with Serverless Inference. For the workloads above, it's the cleanest pricing model available. But that one pricing model isn’t the right answer for every workload.

Four workload patterns, four honest answers

So which of your workloads belong on token pricing, and which have outgrown it? 

Let's look at four patterns production inference tends to settle into. These four examples are directional—no two inference workloads are the same—but holding your own up against them is a useful exercise to see which pricing model could fit.

Three inference paths, one stack

The practical expression of this framework is CoreWeave Inference, with different paths to deployment sitting on top of the same purpose-built infrastructure, offering a pricing model for every workload  and deep observability at every layer.

• Serverless Inference is best for iteration, variable workloads, and production patterns where token pricing is the honest lens. Zero capacity management; fast path from prototype to live traffic.
• Dedicated Inference is ideal for SLO-bound production with predictable traffic. Reserved capacity, tail-latency guarantees, GPU-billed economics aligned to sustained utilization.

A third path, Inference on CKS, covers the edge cases that need full Kubernetes control, strictly regulated, multimodal, or ultra-high-scale workloads, on the same GPU-billed economics as Dedicated Inference with additional capacity options. Different parts of a mature AI product belong on different paths. A customer-facing assistant might run on Dedicated Inference; the experimental agent team might run on Serverless Inference; a regulated enterprise tier might run Inference on CKS. With CoreWeave Inference, different parts of the same product can  live on the same infrastructure, share the same observability, and you can scale up or down between paths without re-platforming.

You don't pick a pricing model for your whole AI program. You pick the one that fits each workload—and adapt it as the workload changes.

What to ask your inference provider

When you evaluate inference, ask these questions of any provider you're evaluating, and be realistic about your own stack. The answers tell you where each workload belongs today and what scaling looks like.

• Can you compute performance-adjusted cost per useful token for this workload on each of your pricing models?
• What are my p95 and p99 time-to-first-token under a 3× traffic burst?
• At what sustained utilization does dedicated capacity beat token pricing for my workload shape?
• Can I mix pricing models across workloads on the same platform?
• Can I move a workload between paths as it matures, without re-platforming?

Mature AI platform teams don't pick one pricing model across the board. They pick the right one for each workload, and change it when the workload changes. 

Get a performance-adjusted TCO analysis of your workload

Schedule a TCO consultation and our team will benchmark each of your workloads against the CoreWeave Inference deployment paths. You’ll get a recommendation on which pricing model and path fits each workload today, and what graduation looks like as it scales.

Want to learn more about TCO for AI infrastructure? Read our blog or check out Signal65’s comprehensive TCO analysis of AI cloud deployments