Fault tolerance: timeouts, retries, circuit breakers

Fault tolerance means the system keeps working, perhaps in a reduced way, when parts of it fail.

Why it matters

In a system of many services, something is always slow or broken. Without protection, one slow dependency makes its callers wait, they run out of threads, and the failure spreads until everything is down.

The idea

In distributed systems something is always failing: a server, a network link, a slow dependency. There are five defences. Timeouts: never wait forever. Retries with exponential backoff and jitter: try again, gently. Circuit breakers: stop calling a broken service for a while. Fallbacks: return cached or default data. Bulkheads: separate resources, so one failure can't sink everything.

A fuse box. When one circuit overloads, its fuse trips, so the whole house doesn't catch fire.

A fuse stays off until someone resets it. A circuit breaker tests the service again on its own after a cool-down (the 'half-open' state) and closes again when calls succeed.

How it works

  1. Timeout: decide how long a call may take, say 300 ms. After that, give up instead of waiting forever.
  2. Retry: if the failure looks temporary and the call is safe to repeat, try again after 100 ms, then 200 ms, then 400 ms (exponential backoff), each with a little randomness (jitter).
  3. Circuit breaker: count failures. If, say, half the calls fail within 10 seconds, 'open' the breaker and fail instantly for the next 30 seconds without calling the service at all.
  4. Half-open: after the wait, let a few test calls through. If they succeed, close the breaker; if not, open it again.
  5. Fallback: while the breaker is open, return something useful: cached data, a default, or the page without that feature.

Example

Checkout calls a recommendations service. If it's slow, a 200 ms timeout plus a circuit breaker lets checkout still load, just without recommendations.

Key points

Common mistakes

Not true: Retries: a failed card payment can be sent again like a failed read.
In fact: The first attempt may have worked before its reply was lost, so a retry could charge the card twice. Retry only operations that are safe to repeat.

Not true: During an outage: retry at once, with no delay, to recover sooner.
In fact: Instant retries from many clients pile more load onto a service that is already struggling. Wait longer after each failure (exponential backoff), with some random jitter.

In the interview

Every network call gets a timeout. Idempotent calls are retried with exponential backoff and jitter. Calls to non-critical dependencies sit behind a circuit breaker with a fallback, so the core feature keeps working when they fail. That stops one slow service from causing a cascading failure.

Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.