Your CFO asked you to budget for LLM API calls. You estimated based on test data: 1,000 requests per day, $0.05 per request. Call it $50 a day. Maybe $1,500 per month.
Then you shipped to production.
By month two, your bill was $45,000. By month four, the board was asking questions.
This isn’t an anomaly. It’s the most predictable failure mode in AI cost planning. And it almost always happens the same way.
Where the Math Breaks
Your production traffic is different from your demo traffic.
Your demo probably ran a handful of carefully constructed test cases. Production data is noisier, more edge-casey, and requires more context than you expected. That changes the token count immediately. Not by 10%. Often by 2–3x.
Then there’s the invisible multiplication: multi-step workflows, retries, clarification requests, fallback logic, token overhead from chunking and reranking.
A single user request that looks like one API call is often:
- One embedding call to find relevant context (1,000 tokens)
- One retrieval call to get the data (500 tokens)
- One reranking call to filter results (1,000 tokens)
- One main inference call (3,000 tokens input, 1,500 tokens output)
- One validation call to check the response (2,000 tokens)
- One fallback call if the first response wasn’t good enough (3,000 tokens)
That single “request” consumed 12,000 tokens. At $0.01 per 1,000 tokens on a frontier model, that’s $0.12 per user request. With error handling and retries at scale, you’re closer to $0.15–$0.20.
Most teams budget for $0.05 and get shocked at $0.15.
The Invisible Multipliers
The math above assumes production-efficient systems. Most aren’t, early on.
Overfitting to prompts. You optimize one prompt on your internal test set, then ship it. The first week of production reveals it hallucinates on real user data. So you build a more robust prompt. More context. More examples. More tokens.
Retry loops. Production has failures your test data didn’t predict. Rate limits, timeouts, edge cases that make the model fail. You add retry logic. One failure becomes two API calls. Two becomes three if the first retry also fails.
Context bloat. You start with a reasonable context window. Then you realize it’s missing edge cases. So you add more context. Then more. By month three, you’re sending 5x the context you planned for, and the model spends 70% of its tokens just reading instead of reasoning.
Evaluation overhead. You need to measure whether the system works. So you sample responses and run them through a separate evaluation model. Another 1,000–2,000 tokens per inference. Multiply by your throughput.
Hallucination defense stacking. The model generates false confidences on 5% of requests. You add guardrails — another model call to validate outputs. Then a semantic check. Then a rule-based check. Pretty soon you spend as many tokens on validation as on actual inference.
Each decision makes sense on its own. Together, they multiply token consumption by 5–10x.
The Real Budget
Here’s what actually-shipped AI systems look like from a cost perspective:
- Base inference: 30–40% of tokens
- Context retrieval and preparation: 25–35%
- Evaluation and monitoring: 15–20%
- Retries and error handling: 10–15%
Your “base” inference might cost $0.05 per request. But all-in, you’re at $0.20–$0.30.
At 10,000 requests per day, that’s $2,000–$3,000 daily. Not $500.
The Cost Control Decision
Teams that don’t get surprised make explicit cost-control decisions early.
Token budget by component. How many tokens can retrieval use? Inference? Validation? Set explicit ceilings and stick to them.
Tiered model routing. Don’t run every request through your most expensive frontier model. Route 70% through a faster, cheaper model and reserve the frontier model for when you need it. That alone can cut costs by 60–70%.
Batch processing where possible. If you don’t need real-time responses, batch overnight. Orders of magnitude cheaper.
Caching intent and context. Don’t recompute embeddings and retrieval for the same query twice. Cache aggressively.
Measurement before scale. Get your per-request cost accurate before you scale. Most teams scale first and discover the cost structure later.
What This Means
If you’re planning AI in production, do this now:
- Calculate true per-request cost with your actual workflow — not just base inference
- Build cost attribution into your system from week one. Measure by component, not just total tokens
- Set cost limits per feature and per workflow, then enforce them
- Plan for 3–5x your initial estimate when budgeting
The teams that stay profitable treat cost as a first-class architectural concern — not something to figure out after launch.
Ready to get your AI costs under control before they spiral? Let’s talk.