Observability: logs, metrics and traces
Observability is how well you can understand what a system is doing from the outside, using logs, metrics and traces.
Why it matters
When something goes wrong in a system of many services, you can't attach a debugger to production. Without data coming out of the system, you're guessing.
The idea
Metrics are numbers over time (requests per second, error rate, latency percentiles), used for dashboards and alerts. Logs are detailed records of events, used for investigating. Traces follow one request across many services and show where the time went. Alert on symptoms users feel (errors, latency against your SLO), not on every CPU blip.
A car: the dashboard is metrics, the service history is logs, and a GPS replay of one journey is a trace.
A car is one machine. A request may cross 20 services, so without a shared trace ID to join the logs together, you can't follow it.
How it works
- Metrics: every service counts requests, errors and latency, and exports them every few seconds. Dashboards show trends; an alert fires when, say, the error rate stays above 1% for 5 minutes.
- Logs: each important event is written as structured JSON, including the request's trace ID, the user ID and timings.
- Traces: the first service creates a trace ID and passes it in a header to every service it calls. Each service records how long its part took.
- Investigating: the alert tells you something is wrong, the dashboard shows when it started, a trace shows which service is slow, and that service's logs show why.
Example
The p99 checkout latency jumps from 300 ms to 2 s. The dashboard shows when; a trace shows 1.6 s spent in the payment service's database call; that service's logs show a lock timeout.
Key points
- The four golden signals: latency, traffic, errors, saturation.
- Use percentiles (p50, p95, p99), not averages; averages hide the slow users.
- Structured logs (JSON), with a request or trace ID on every line.
- Distributed tracing (OpenTelemetry) shows the time spent in each service.
- Alert on SLO burn (user impact), and give every alert a runbook.
- Health checks and a dashboard per service.
Common mistakes
- Alerting on causes (CPU at 80%) instead of symptoms users feel (errors, slow responses).
- Logs without a request or trace ID, so you can't connect them.
- Watching the average latency instead of percentiles: an average hides the slow requests that users actually feel.
Not true: The best alerts fire on causes, such as CPU reaching 80%.
In fact: High CPU may hurt nobody, and users can suffer while CPU looks fine. Alert on the symptoms users feel: errors and slow responses.
Not true: Average latency shows what the slowest users experience.
In fact: An average hides the slow requests. Percentiles show them: p99 is the time that 99% of requests beat, so it describes the slowest 1%.
In the interview
I'd emit the four golden signals as metrics for each service, write structured logs with a trace ID, and use distributed tracing to follow requests across services. Alerts fire on user-facing symptoms, like p99 latency or error rate breaking the SLO, so on-call engineers are paged only for real problems.
Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.