Caching, model routing and cost control
Cost control for LLM apps uses caching (reuse answers or processed prompts), routing (send easy requests to cheaper models) and budgets (limits, timeouts and per-request cost tracking).
Why it matters
LLM costs scale with every token and every user, and the biggest model is often far more than most requests need. Without control, a successful product can become unaffordable, and slow. Caching and routing commonly cut costs by large factors with little or no loss in quality, if you measure that quality.
The idea
LLM apps can get slow and expensive fast. There are three main tools. Caching reuses answers, or processed prompts, you have already paid for. Routing sends easy requests to a small, cheap model and only hard ones to a big model. Budgets limit tokens, set timeouts and track the cost of every request.
A help desk. Common questions get a ready-made answer sheet (the cache). Simple questions go to a junior (the small model), hard ones to a senior (the big model).
A help desk knows when two questions are really the same. A semantic cache matches questions by meaning, so it can return a wrong answer for a question that only looks similar. Use a strict similarity threshold and an expiry time.
How it works
- Exact cache: store answers keyed by the exact request; a repeat costs nothing.
- Semantic cache: match new questions to stored ones by embedding similarity, with a strict threshold and an expiry time.
- Prompt caching: providers can reuse the processed form of a long, unchanging prompt prefix, making repeated calls cheaper and faster. Keep stable content (instructions, documents) at the start.
- Routing: a rule or small classifier sends easy requests to a small model and hard ones to a large model.
- The saving from routing is roughly (share routed) × (1 − cost ratio): 70% of traffic at a tenth of the price saves about 63%.
- Budgets: cap tokens per request, set timeouts, and log the cost of every request.
- Check quality on your eval set before and after, so savings don't hide worse answers.
Key points
- Cache what repeats; route what is easy; cap what is unbounded.
- Semantic caches can return wrong answers for questions that only look similar.
- Prompt caching rewards stable prompt prefixes.
- Track cost per request and per user.
- Every cost optimisation needs a quality check.
Common mistakes
- A semantic cache threshold so loose that different questions share answers.
- Caching personalised or time-sensitive answers, so one user is shown another's answer, or yesterday's facts.
- Routing hard questions to the small model without measuring.
- Putting changing content at the start of the prompt, which defeats prompt caching.
Not true: Routing only affects cost, so hard questions can go to the small model without checking.
In fact: Routing affects quality too: a hard question sent to a weak model gets a worse answer. Measure the quality of each route before relying on it.
Not true: A semantic cache can safely store any answer, including personal ones.
In fact: A cached personal answer can be shown to a different user, and a cached fact can go out of date. Cache only what is the same for everyone and slow to change.
Part of Unit 5: AI engineering in the free ML Foundations course.