Article
The platform served dozens of tenants through one shared set of AI features, and the monthly model bill had tripled in a quarter with no corresponding growth in revenue. The cause was not inefficiency in general; it was three specific, fixable patterns, and finding them required looking at cost per tenant, not cost in aggregate, because the aggregate number hid which tenants were actually expensive and why.
The first finding: one tenant was 40% of the bill
Aggregate cost dashboards answer "how much are we spending," not "who is spending it and on what." Once we broke the bill down per tenant, per feature, and per model, one tenant's usage of a single feature — a document summarizer being called on entire multi-hundred-page files instead of sections — accounted for close to half the platform's total spend. No amount of general optimization would have found this; it required per-tenant observability that did not exist yet.
Fix one: chunk-aware limits, not blanket rate limits
The immediate fix was not to throttle the tenant, which would have degraded a paying customer's experience, but to cap the input size per summarization call and chunk longer documents into sections summarized separately and then combined. This cut that feature's per-call cost by roughly 70% with no perceptible quality loss, because the model was previously re-reading hundreds of pages of stable content on every call instead of processing what had actually changed.
Fix two: routing by task, not by tenant plan
The platform had been sending every request through the same large model regardless of task, on the reasoning that it was the safest default. Once we routed by request type — classification and extraction to a small, fine-tuned model, and only genuinely complex synthesis to the large one — roughly 65% of total request volume moved to a model costing a fraction as much per call, cutting cost on that volume by over 80%.
Fix three: prompt caching across tenants, done carefully
Several features shared a large, stable system prompt — instructions, schema definitions, examples — with only the tenant's specific data changing per call. Restructuring prompts so the stable portion came first, cacheable, and the tenant-specific portion came last, let the model provider's prompt caching discount apply to nearly every call, cutting the token cost on those features by roughly a third. The care required was making sure no tenant-specific data ever leaked into the cached, shared prefix — a mistake there is a data isolation bug, not just a cost one, and it is the same discipline as tenant isolation anywhere else in the platform.
Fix four: a budget per tenant, with a graceful degrade
The longer-term fix was structural: every tenant got a daily cost ceiling, and hitting it degraded gracefully — falling back to the small model or queuing non-urgent requests — rather than continuing to bill silently or failing outright. This turned "which tenant is about to cost us money this month" from a surprise into an alert, and gave the account team a real conversation to have before the invoice, not after it.
The result
Combined, these four changes cut the platform's total running cost by 58% within two billing cycles, with the eval set showing quality unchanged on three of the four affected features and slightly improved on the fourth, because the tuned small model was better at the narrow task than the general model had been. None of the individual fixes were exotic; the work was in finding where the cost actually lived, which required tenant-level visibility the platform was not built with from day one. It is now part of how we architect every AI/ML Development platform from the start, rather than retrofitting it after the first surprising invoice.