LLM Cost Control: Tokens, Caching, and Budgets Explained

LLM spend has a habit of growing faster than the product that generates it. The bill is not one number you can tune — it is the sum of thousands of design decisions: how long your system prompt is, how much history you replay, which model handles which task, and whether anything stops a runaway loop at 3 a.m. This post walks through the cost model from first principles and the controls that hold up in production.

Token economics 101

Every request has two meters running: input tokens (everything you send) and output tokens (everything the model generates). They are priced differently, and the asymmetry matters. Output tokens typically cost 3–5x more than input tokens, because generation is sequential and compute-heavy while input processing is parallelizable.

That asymmetry leads people to obsess over output length, but for most products the input side dominates in volume. A typical chat or agent request might carry 8,000 input tokens — system prompt, tool definitions, retrieved context, conversation history — to produce 300 output tokens. Even at a 4x output multiplier, input is still 85 percent of the cost of that request.

The biggest single line item is usually the system prompt, because it is paid on every request. A 3,000-token system prompt served a million times a day is three billion input tokens per day before a single user word is processed. When you review prompts, review them like you review hot-path code: every paragraph in the stable prefix is a recurring cost, not a one-time one.

Prompt caching

Most major providers now offer prompt caching: if the beginning of your request is byte-identical to a recent request, the provider reuses the precomputed attention state for that prefix instead of reprocessing it, and charges a fraction of the normal input price — typically anywhere from 10 to 50 percent of the normal price depending on the provider, sometimes with a small write surcharge on the first request.

The key mechanic is that caching is prefix-based. The provider matches from the first token forward and stops at the first difference. That dictates how you should structure prompts:

1. Stable content first. System prompt, tool schemas, few-shot examples — anything identical across requests goes at the top, in a fixed order.
2. Semi-stable content next. Per-user or per-session context that changes occasionally.
3. Volatile content last. The current user message, timestamps, request IDs, retrieved documents.

The classic cache-killer is a dynamic value near the top of the prompt. A timestamp, a random session ID, or a "today's date" line in the first hundred tokens invalidates the entire cache for everything after it. Move it to the end, or drop it. Also watch serialization order: if your tool definitions come from an unordered map, they may serialize in a different order per process and silently destroy your hit rate. In our experience running production LLM traffic, reordering alone — with no prompt content changes at all — can take cache hit rates from under 20 percent to over 90 percent.

Model routing for cost

Flagship models are priced 10–60x above small models from the same provider. Using a frontier model to classify a support ticket into five categories is like renting a GPU cluster to run grep. Right-sizing means mapping each task to the cheapest model that meets its quality bar, measured on your own eval set rather than leaderboard vibes.

A step beyond static assignment is the cascade pattern: send the request to a cheap model first, and escalate to a stronger model only when needed. The escalation trigger can be explicit (the small model outputs a low confidence score or an "unsure" token), structural (its output fails schema validation), or downstream (the user hits retry). If the cheap model handles 80 percent of traffic and the expensive model costs 20x more, the cascade cuts spend by roughly 75 percent versus routing everything to the flagship — and the p50 latency usually improves too.

Cascades add a failure mode worth naming: if your escalation check is too loose, you pay for both calls on most requests and save nothing. Instrument the escalation rate as a first-class metric. Running this through a gateway that speaks one API across 20+ models makes the experiment cheap, because rerouting is a config change instead of a code change.

Budgets and alerts

Cost control without enforcement is a dashboard, not a control. You want budgets at two granularities:

• Per-key budgets cap what a single API key can spend per day or month. Every service, environment, and external integration should have its own key. A leaked or looping key then has a bounded blast radius.
• Per-feature budgets cap a product surface — "AI summaries may spend $200/day" — independent of which keys serve it.

Distinguish soft alerts from kill switches, and use both. A soft alert at 70–80 percent of budget pages a human while there is still time to react. A hard kill switch at 100 percent stops traffic outright. Kill switches feel drastic, but the alternative is discovering a retry loop via the invoice. The practical compromise for user-facing features is graceful degradation at the limit: route to the cheapest viable model, shrink context, or disable the feature with an honest error, rather than failing silently. Whatever you choose, the budget check must be enforced at the gateway or middleware layer — client-side checks do not survive bugs, and bugs are exactly what budgets exist to contain.

Measuring cost per feature and per user

Aggregate spend tells you that you have a problem; attribution tells you where. Tag every request with a feature identifier and a user or tenant identifier — request metadata or key naming conventions both work — and roll tokens up along those dimensions. This turns token usage into a product metric, on the same footing as latency or conversion.

Two derived numbers earn their place on a weekly dashboard:

• Cost per feature per day, which tells you which surfaces to optimize first. It is common to find that one background enrichment job outspends the entire user-facing product.
• Cost per active user per month, which is a unit-economics input. If your plan is priced at $20/month and your p95 user consumes $14 of inference, your margin lives inside the p95 tail — and a handful of power users can make a flat-rate tier unprofitable on their own.

When tracking cache-heavy workloads, record both billed cost and what the same tokens would have cost uncached. The gap is the value of your caching work, and it stops regressions from hiding inside a flat total.

Common waste patterns

A few patterns account for most of the avoidable spend we see:

• Unbounded chat history. Replaying the full conversation on every turn makes per-turn cost grow linearly and total conversation cost grow quadratically with length. Cap the window and summarize older turns; almost no task needs verbatim turn 3 at turn 60.
• Retry amplification. Client retries stacked on SDK retries stacked on gateway retries can turn one failed request into eight billed ones. Retry at one layer, with backoff and a total-attempt cap, and never blind-retry a request that already returned billable output.
• Verbose output formats. Asking for JSON with long, repeated key names, or letting the model narrate before the answer ("Certainly, here is the analysis you requested..."), pays the premium output rate for filler. Use terse schemas, ask for the answer directly, and set max_tokens to a realistic bound per task.
• Oversized retrieval. Stuffing ten retrieved documents into context when the answer needs two. Measure answer quality against context size; the curve flattens earlier than most teams expect.

None of these require research-grade work. They require treating tokens the way you already treat database queries and egress bandwidth: as a metered resource with an owner, a budget, and a graph someone actually looks at. Do the audit once, wire the budgets in, and the bill becomes a design input instead of a monthly surprise.