Observability: tracing and debugging LLM apps

Observability means recording what your LLM app did on every request: prompts, retrieved chunks, tool calls, outputs, tokens, time and cost. The record is a trace you can inspect, search and learn from.

Why it matters

LLM apps fail in quiet, specific ways: the wrong chunk retrieved, a tool error swallowed, an answer cut off. Without traces, a bad answer is a mystery, and you end up tweaking prompts at random. With traces, you can point at the step that went wrong, see where time and money go, and turn real failures into eval cases.

The idea

Observability means you can see what your app actually did for any single request. That covers the prompt, the retrieved chunks, every tool call and the model's reply, plus the tokens, cost and time of each step. A trace records all of this as one tree of steps. Without it, a bad answer is a mystery. With it, you can point at the step that went wrong.

A flight recorder, the 'black box'. When something goes wrong, you replay exactly what happened instead of guessing.

A flight recorder is only read after a crash. LLM traces are most useful every day: you read a sample of real traces, and turn the failures you find into new eval cases.

How it works

  1. Give every request a trace: a tree of steps (spans), such as embed, retrieve, re-rank, LLM call and tool call.
  2. For each span record its inputs, outputs, start and end time, token counts and cost, plus any errors.
  3. Draw traces as a waterfall to see which step dominates latency.
  4. Search and filter traces: slowest, most expensive, user-flagged, failed.
  5. Read a sample of real traces regularly, not only after complaints.
  6. Turn each failure you find into a new eval case.
  7. Redact or avoid logging personal data, and keep traces for a limited time.

Key points

Common mistakes

Not true: Prompts and outputs can be logged in full, personal data included, without any risk.
In fact: Logs are stored, shared and searched, so personal data in them is a real risk. Remove or hide sensitive fields before logging.

Not true: A log of the final answers gives what is needed to debug an LLM app.
In fact: When the answer is wrong, you need to see which step failed: the retrieval, a tool call or the model. Log the inputs and outputs of every step.

Part of Unit 5: AI engineering in the free ML Foundations course.