Distributed transactions: sagas and the outbox pattern

A saga runs a business process across several services as a series of local transactions, with a compensating action to undo each step if a later step fails.

Why it matters

An order needs stock reserved, a card charged and a shipment booked, in three different services with three databases. No single transaction can cover all three, but you still can't leave an order half done.

The idea

Across services you can't wrap everything in one ACID transaction. A saga runs steps one by one — reserve stock, charge the card, create the shipment — and if charging fails, it runs compensations, like releasing the stock. Steps are coordinated by choreography (services react to each other's events) or orchestration (a central coordinator tells each step what to do).

The outbox pattern makes 'update my database and publish an event' reliable: write the event into an outbox table in the same transaction, then publish it.

Booking a holiday: flight, hotel, car. If the car booking fails, you cancel the hotel and the flight. You don't pretend the trip never happened; you undo each step.

When you cancel a holiday, nobody else sees the half-booked state. In a saga, other users can see the in-between state (stock reserved, then released), so every step and compensation must allow for that, and be idempotent.

How it works

  1. The orchestrator starts the saga: 'reserve stock'. The stock service reserves it and replies OK.
  2. Next, 'charge the card'. The payment service tries, and the card is declined.
  3. The orchestrator runs the compensations in reverse order: 'release the stock'.
  4. The order is marked FAILED and the user is told.
  5. Outbox: each service, in the same local transaction as its change, writes an event row into an outbox table. A relay reads new rows and publishes them, retrying until it succeeds.

Example

The order service writes the order and an 'OrderCreated' outbox row in one transaction. A relay publishes it to Kafka. The payment service charges the card, or emits 'PaymentFailed' if it's declined. In that case the order service cancels the order.

Key points

Common mistakes

Not true: Saving to the database and then publishing an event always succeed or fail together.
In fact: A crash between the two leaves the data saved and the event never sent (the dual-write problem). The outbox pattern writes the event in the same transaction as the data.

Not true: Saga: a failed step is undone by the database's own rollback.
In fact: Each step is its own committed transaction, so there is nothing left to roll back. A saga runs a compensating action instead, such as releasing the stock.

In the interview

For checkout across services I'd use an orchestrated saga: reserve stock, take payment, confirm the order, with a compensating step for each. Each service writes its state change and its outgoing event atomically using the outbox pattern, and every handler is idempotent.

Part of Unit 4: Distributed systems and security in the free System Design course.