For founders
Keeping AI feature costs sane in 2027
a16z's analysis found LLM inference cost for equivalent performance falling about tenfold a year: GPT-3 cost $60 per million tokens in 2021; equivalent capability cost cents by late 2024. Your AI feature's bill still grows if the feature is designed wrong. Five habits that keep it below the hosting bill.
Prashant Kumar Singh · · 4 min read
In short
- For equivalent performance, LLM inference cost has fallen roughly 10x per year (a16z). The model is not where the bill comes from.
- The bill grows when the feature sends the whole document every time, retries silently, or runs on every keystroke.
- Five habits: cache repeated prompts, send only the context the job needs, route with a small model and answer with a large one, cap retries, set a per-user budget. Measure cost per completed job, not per token.
New year, same question from every founder planning an AI feature: what will it cost to run? The honest answer has two parts. The model is cheaper than you think, and the feature can still be more expensive than you can afford, depending entirely on how it is built.
The model is getting cheaper fast
In November 2024, a16z published an analysis they called LLMflation: for a fixed level of capability, the cost of LLM inference has been falling by roughly ten times a year. Their anchor: GPT-3 launched in 2021 at about $60 per million tokens; by late 2024, models matching its performance cost around $0.06 per million. A thousandfold drop in three years. Whatever your feature costs to run today, the same capability will cost a fraction of that by the time you have real scale.
So why do AI bills surprise people?
Because the bill is not the model price. It is the model price multiplied by how many tokens the feature sends, how many times, for how many users. Three design mistakes account for most surprises.
- Sending everything, every time. The whole document, the whole chat history, the whole knowledge base, on each request, when the job needed one paragraph.
- Retrying silently. A failed call retried three times, with the full context each time, and nobody counting.
- Running on every keystroke. Autocomplete-style features that call the model on each character instead of on a pause or a submit.
Five habits that keep it sane
- 1.Cache repeated prompts. The system prompt, the product's instructions and the shared context are the same for every user. Most providers cache them at a steep discount; use it.
- 2.Send only the context the job needs. Retrieve the relevant passage, not the document. Summarise the history, do not replay it. Context is the main cost lever.
- 3.Route small, answer large. A small, cheap model decides what kind of request this is and whether a model is needed at all. The large model answers only the requests that need it.
- 4.Cap retries and time them. One retry, with backoff, and a log line. A feature that retries without a cap will find a way to spend the month's budget in an afternoon.
- 5.Set a per-user budget. A ceiling on tokens per user per day, with a graceful message when it is hit. Abuse and bugs both stop at the ceiling.
Measure the right number
Cost per token is the provider's number. Cost per completed job, a draft written, a document summarised, a question answered, is yours. It is the number your pricing has to cover and the number that tells you whether habit two is working. Put it on the dashboard next to the feature's usage, and watch it fall as the model prices fall and the context gets tighter.
| Habit | Typical saving | Effort |
|---|---|---|
| Prompt caching | Large on the fixed part of every call | An afternoon |
| Context trimming | The biggest lever; often the majority of tokens | Ongoing design discipline |
| Small-model routing | Large models used only where needed | A day |
| Retry cap | Prevents the worst months | An hour |
| Per-user budget | Prevents abuse and runaway bugs | An hour |
A note on model choice
The temptation is to pick the cheapest model. The better move is to pick the model that gets the job right on the first call, because a wrong answer that the user retries, or that you retry, costs more than a right one from a more expensive model. Price per million tokens is a distraction; price per correct completed job is the metric, and the tenfold-a-year trend means it is falling for every model on the list.
Questions people ask
- How fast are LLM costs falling?
- a16z's LLMflation analysis found inference cost for a fixed level of capability falling about tenfold per year. GPT-3 cost around $60 per million tokens in 2021; equivalent capability cost about $0.06 per million by late 2024.
- Why is my AI feature expensive to run?
- Usually design, not model price: sending the whole document or history on every call, retrying failed calls without a cap, or calling the model on every keystroke. Trimming context is the biggest single saving.
- How do you control the cost of an AI feature?
- Cache the fixed part of prompts, send only the context the job needs, route requests with a small model and answer with a large one only when needed, cap retries, set a per-user budget, and measure cost per completed job rather than per token.
If you're a founder
Planning an AI feature for 2027? Send the plan and the free teardown will include a cost sketch: tokens per job, the habits that apply, and what the feature should cost per user.
If you run an AI lab or data team
Cost per completed job is the same idea as cost per accepted task: the unit that matters is the one you can use, not the one the invoice counts.
Sources