Article
The prototype used the most capable model for every request, because during a demo, cost is nobody's problem. Then the feature ships, usage climbs, and the monthly bill becomes the first thing the finance team asks about. The fix is not to use a worse model. It is to stop using the best model for work that does not need it.
Route by difficulty
Most requests to an AI feature are easy. A classification, a short extraction, a rephrase, a yes-or-no check. A small model handles these at a small fraction of the cost and often faster. The hard requests, the ones that need reasoning across several documents or a long, careful answer, go to the large model. A router decides which is which.
The router itself can be a small model, a set of rules, or both. Ours usually starts as rules (request length, task type, whether retrieval found anything) and gains a small classifier once we have data. The important thing is that the routing decision is logged, so we can see what went where and whether the evaluation set agrees with the choice.
Cache what repeats
Two kinds of cache matter. Prompt caching, where the model provider charges much less for a prefix it has seen recently: put the stable parts of the prompt, such as instructions and reference material, at the front, and the variable parts at the end, and the same content stops being paid for on every request. And response caching for genuinely identical requests, which are more common than people expect once a feature is used by many people asking the same things.
Make the small model good enough
Small open-weight models running in your own cloud are now capable of most classification and extraction work, and for a narrow task, a fine-tuned small model often beats a large general one. Fine-tuning needs a few hundred good examples, which the evaluation set already provides. This is where the numbers get dramatic: a task that cost cents per request on a frontier model can cost hundredths of a cent on a tuned small model, with a higher score on your data.
Cut the tokens, not the quality
Shorter prompts, tighter retrieval, and summarised tool results reduce cost on every model. Half of the "expensive" AI features we are asked to look at are sending three times more context than the answer needs. The context engineering that improves accuracy also cuts the bill.
Put a budget on every feature
Each AI feature gets a cost ceiling per request and per day, with alerts. When a limit is hit, the feature degrades gracefully, for example by falling back to the small model or asking the user to narrow the request, rather than silently running up a bill. This is the control the finance team actually wants, and it takes an afternoon to build.
What we typically see
On the integrations we have done, routing and caching alone have cut running cost by roughly half against the prototype, and a tuned small model for the high-volume path has taken it further. Quality, measured on the test set, went up more often than down, because the small model was tuned on the client's data and the large model was reserved for the requests that needed it.
None of this is exotic. It is the same discipline as any other production system: measure, budget, route, cache. The models are new; the engineering is not.