Serving: APIs, Docker, latency and cost
Serving puts a model behind an API so applications can use it, packaged (often in a Docker container) with everything it needs, and measured for latency, throughput and cost.
Why it matters
A model in a notebook helps nobody. Real users need a reliable endpoint that answers quickly, handles many requests at once, and doesn't cost more than it earns. Serving is where your model meets real traffic, and where latency and cost decisions are made.
The idea
Put your model behind an API (for example with FastAPI), package it with everything it needs (Docker), and measure latency (time per request) and cost (per request or per token).
A restaurant: the recipe is the model, the serving counter is the API, and a food truck you can drive anywhere is the Docker container.
A chef cooks one order at a time. Model servers batch many requests together, which serves more users but can make each one wait a little longer.
How it works
- Wrap the model in a web API (for example FastAPI): an endpoint like POST /generate that validates input and returns output.
- Add a /health endpoint, request IDs and structured logs (latency, tokens, errors) for every call.
- Package the code, model files and dependencies into a Docker image, so it runs the same everywhere.
- Measure latency: time to first token (what users feel first) and tokens per second after that.
- Under load, batch requests: one GPU pass serves many users. Continuous batching (vLLM, TGI) adds new requests as old ones finish.
- Stream tokens back as they are generated, so users see progress immediately.
- Estimate cost per request from input and output tokens, or from GPU hours and throughput, and track it.
Key points
- API + container + monitoring = a deployable service.
- Time to first token and tokens per second are the key latency numbers.
- Batching trades a little latency per user for much higher throughput.
- Streaming improves the speed people feel: the first words appear at once, although the full answer takes just as long.
- Output tokens usually cost more than input tokens.
- Load-test before launch: send many requests at once to find the limit before real users do.
Common mistakes
- Only testing one request at a time.
- No timeouts or limits, so a slow request blocks others.
- Logging prompts that contain personal data without care.
- Ignoring cold-start time when loading big models.
Not true: A service that answers one request quickly will handle a thousand at once as quickly.
In fact: Under load, requests queue up, memory fills and response times climb. Send many requests at the same time before launch, to find the limit.
Not true: A service does not need timeouts: slow requests finish in the end.
In fact: One slow request can hold a worker or a GPU slot while others queue behind it. Set timeouts and limits, so that a stuck request is cut off.
Part of Unit 5: AI engineering in the free ML Foundations course.