Observability: logs, metrics and traces

Observability is how well you can understand what a system is doing from the outside, using logs, metrics and traces.

Why it matters

When something goes wrong in a system of many services, you can't attach a debugger to production. Without data coming out of the system, you're guessing.

The idea

Metrics are numbers over time (requests per second, error rate, latency percentiles), used for dashboards and alerts. Logs are detailed records of events, used for investigating. Traces follow one request across many services and show where the time went. Alert on symptoms users feel (errors, latency against your SLO), not on every CPU blip.

A car: the dashboard is metrics, the service history is logs, and a GPS replay of one journey is a trace.

A car is one machine. A request may cross 20 services, so without a shared trace ID to join the logs together, you can't follow it.

How it works

  1. Metrics: every service counts requests, errors and latency, and exports them every few seconds. Dashboards show trends; an alert fires when, say, the error rate stays above 1% for 5 minutes.
  2. Logs: each important event is written as structured JSON, including the request's trace ID, the user ID and timings.
  3. Traces: the first service creates a trace ID and passes it in a header to every service it calls. Each service records how long its part took.
  4. Investigating: the alert tells you something is wrong, the dashboard shows when it started, a trace shows which service is slow, and that service's logs show why.

Example

The p99 checkout latency jumps from 300 ms to 2 s. The dashboard shows when; a trace shows 1.6 s spent in the payment service's database call; that service's logs show a lock timeout.

Key points

Common mistakes

Not true: The best alerts fire on causes, such as CPU reaching 80%.
In fact: High CPU may hurt nobody, and users can suffer while CPU looks fine. Alert on the symptoms users feel: errors and slow responses.

Not true: Average latency shows what the slowest users experience.
In fact: An average hides the slow requests. Percentiles show them: p99 is the time that 99% of requests beat, so it describes the slowest 1%.

In the interview

I'd emit the four golden signals as metrics for each service, write structured logs with a trace ID, and use distributed tracing to follow requests across services. Alerts fire on user-facing symptoms, like p99 latency or error rate breaking the SLO, so on-call engineers are paged only for real problems.

Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.