Consistency, CAP and PACELC

Consistency (in distributed systems) means every read sees the latest write. The CAP theorem says that during a network partition, a system must choose between consistency and availability.

Why it matters

To survive failures and be fast for users everywhere, you keep copies of data on several machines. But copies can't all change at exactly the same instant, and the network between them can break. You must decide what users see in that gap.

The idea

When data is copied across machines, the copies can briefly disagree. Strong consistency makes every read return the latest value, but may have to refuse requests when machines can't talk to each other. Eventual consistency always answers, but a read can be briefly out of date.

Network partitions will happen, so the real choice is C or A during a partition. PACELC adds: else (normally), you trade latency against consistency.

Two bank branches whose phone line is cut. Either they stop withdrawals until the line is back (consistency), or they keep serving customers and reconcile later, risking an overdrawn account (availability).

A bank makes this choice once, as a policy. Real systems choose per operation: a social feed can be eventually consistent while the payment inside the same app must be strongly consistent.

How it works

  1. Picture two replicas, in London and Paris, each holding a balance of £100.
  2. The network between them breaks: a partition. A user in London withdraws £80.
  3. CP choice: London refuses, or only accepts if it can confirm with Paris. Always correct, but some requests fail until the network heals.
  4. AP choice: London accepts and records £20, while Paris still says £100. Both keep answering, but a second £80 withdrawal in Paris would overdraw the account. When the network heals, the system must reconcile the two.
  5. With no partition (normal running), waiting for both copies before confirming is consistent but slower. That's the latency-versus-consistency trade-off in PACELC.

Example

Bank balance: strong consistency (CP). Likes on a post: eventual consistency (AP); nobody minds seeing 1,203 instead of 1,205 for a second.

Key points

Trade-offs

Common mistakes

Not true: With enough replicas, a system can stay both consistent and available during a network partition.
In fact: A replica that is cut off must either refuse requests (consistent) or answer with data that may be old (available). No number of replicas removes that choice.

Not true: CAP is one choice, made once for the whole system.
In fact: Different data needs different choices. A bank balance may need consistency, while a count of likes can favour availability, in the same system.

In the interview

It depends on the data. For money or stock levels I'd choose strong consistency, accepting that some requests fail during a partition. For feeds, likes or view counts I'd choose availability with eventual consistency, because a slightly stale number is harmless and the system stays up.

Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.