A client asked us why their AI bill had tripled in a quarter while usage had only doubled. The answer was in the logs: the same forty questions, asked in slightly different words, thousands of times a day.
What we shipped
- A semantic cache in front of the support assistant, keyed on the meaning of the question rather than its exact wording.
- Per-feature budget alerts, so a spike points at a feature instead of an account.
What broke
The first version of the cache was too generous and answered a billing question with a shipping answer. We tightened the similarity threshold and added the product area to the cache key.
What we'd do differently
Tag every model call with the feature that made it from day one. It costs nothing to add early and a week to reconstruct later.