Design an LLM serving platform
Serve large language models to many applications with low latency and cost.
Why interviewers ask it
It tests modern AI infrastructure: GPU cost, batching, memory limits, streaming, and fairness between teams.
Functional requirements
- A chat API with streaming responses.
- Several models and versions; quotas per team.
- Usage metering.
Non-functional requirements
- Time to first token under about 1 second, then a steady stream of tokens.
- High GPU utilisation; graceful behaviour under overload.
- Isolation between teams.
From version 1 to the final design
- Version 1: one model on one GPU server behind an API, handling one request at a time.
- Version 2: static batching, then continuous batching with a paged KV cache.
- Version 3: a gateway with quotas and routing, and autoscaled GPU pools per model.
- Version 4: quantised tiers, prefix caching, speculative decoding and priority queues.
The design, step by step
- A gateway handles authentication, quotas, rate limits and routing to a pool of servers per model.
- Model servers (vLLM-style) use continuous batching and a paged KV cache to keep the GPUs busy.
- GPU pools scale automatically on queue length and latency.
- Responses stream back through the gateway; prefix caching reuses work for repeated system prompts.
- Metrics: time to first token, tokens per second, GPU utilisation, errors, and usage per team.
Deep dives
- Prefill (reading the prompt) and decode (generating one token at a time) have different bottlenecks.
- Continuous batching adds new requests between decode steps instead of waiting for a whole batch to finish.
- Quantised, smaller or distilled models for cheaper tiers; speculative decoding for speed.
- Overload: a bounded queue, and dropping low-priority traffic first.
Common mistakes
- Handling one request at a time on an expensive GPU.
- Ignoring KV-cache memory when planning capacity.
- No per-team quotas, so one team can use it all and leave nothing for the others.
Not true: Handling one request at a time makes full use of an expensive GPU.
In fact: A GPU can work on many requests in the same pass. Continuous batching adds new requests between decode steps, which raises throughput a great deal.
In the interview
A gateway handles auth, quotas and routing to a GPU pool per model. Each replica runs an inference server with continuous batching and a paged KV cache to keep the GPUs busy, and streams tokens back. Pools scale on queue length and latency, and we track time to first token, tokens per second and cost per team. Quantisation, prefix caching and speculative decoding cut cost and latency.
Part of Unit 9: AI and ML system design in the free System Design course.