Design an LLM serving platform

Serve large language models to many applications with low latency and cost.

Why interviewers ask it

It tests modern AI infrastructure: GPU cost, batching, memory limits, streaming, and fairness between teams.

Functional requirements

Non-functional requirements

From version 1 to the final design

  1. Version 1: one model on one GPU server behind an API, handling one request at a time.
  2. Version 2: static batching, then continuous batching with a paged KV cache.
  3. Version 3: a gateway with quotas and routing, and autoscaled GPU pools per model.
  4. Version 4: quantised tiers, prefix caching, speculative decoding and priority queues.

The design, step by step

  1. A gateway handles authentication, quotas, rate limits and routing to a pool of servers per model.
  2. Model servers (vLLM-style) use continuous batching and a paged KV cache to keep the GPUs busy.
  3. GPU pools scale automatically on queue length and latency.
  4. Responses stream back through the gateway; prefix caching reuses work for repeated system prompts.
  5. Metrics: time to first token, tokens per second, GPU utilisation, errors, and usage per team.

Deep dives

Common mistakes

Not true: Handling one request at a time makes full use of an expensive GPU.
In fact: A GPU can work on many requests in the same pass. Continuous batching adds new requests between decode steps, which raises throughput a great deal.

In the interview

A gateway handles auth, quotas and routing to a GPU pool per model. Each replica runs an inference server with continuous batching and a paged KV cache to keep the GPUs busy, and streams tokens back. Pools scale on queue length and latency, and we track time to first token, tokens per second and cost per team. Quantisation, prefix caching and speculative decoding cut cost and latency.

Part of Unit 9: AI and ML system design in the free System Design course.