Consistency, CAP and PACELC
Consistency (in distributed systems) means every read sees the latest write. The CAP theorem says that during a network partition, a system must choose between consistency and availability.
Why it matters
To survive failures and be fast for users everywhere, you keep copies of data on several machines. But copies can't all change at exactly the same instant, and the network between them can break. You must decide what users see in that gap.
The idea
When data is copied across machines, the copies can briefly disagree. Strong consistency makes every read return the latest value, but may have to refuse requests when machines can't talk to each other. Eventual consistency always answers, but a read can be briefly out of date.
Network partitions will happen, so the real choice is C or A during a partition. PACELC adds: else (normally), you trade latency against consistency.
Two bank branches whose phone line is cut. Either they stop withdrawals until the line is back (consistency), or they keep serving customers and reconcile later, risking an overdrawn account (availability).
A bank makes this choice once, as a policy. Real systems choose per operation: a social feed can be eventually consistent while the payment inside the same app must be strongly consistent.
How it works
- Picture two replicas, in London and Paris, each holding a balance of £100.
- The network between them breaks: a partition. A user in London withdraws £80.
- CP choice: London refuses, or only accepts if it can confirm with Paris. Always correct, but some requests fail until the network heals.
- AP choice: London accepts and records £20, while Paris still says £100. Both keep answering, but a second £80 withdrawal in Paris would overdraw the account. When the network heals, the system must reconcile the two.
- With no partition (normal running), waiting for both copies before confirming is consistent but slower. That's the latency-versus-consistency trade-off in PACELC.
Example
Bank balance: strong consistency (CP). Likes on a post: eventual consistency (AP); nobody minds seeing 1,203 instead of 1,205 for a second.
Key points
- C = every read sees the latest write; A = every request gets a non-error response; P = the system keeps working when messages between machines are lost.
- Partitions are unavoidable in distributed systems, so it's CP or AP during a partition.
- Consistency levels: strong, read-your-own-writes, monotonic reads, eventual.
- Quorums: with N copies, writing to W and reading from R where W + R > N means every read overlaps the latest write.
- PACELC: even without partitions, stronger consistency costs latency.
- CAP's C is not ACID's C (which is about the database's rules and constraints).
Trade-offs
- CP: correct, but may reject requests during a partition (bank ledger).
- AP: always answers, but may return stale data and needs conflict handling (shopping cart, likes).
Common mistakes
- Calling a system 'CA'. With replicas, partitions can't be avoided.
- Treating CAP as one choice for the whole system, instead of a choice per type of data.
- Confusing CAP's consistency with ACID's consistency.
Not true: With enough replicas, a system can stay both consistent and available during a network partition.
In fact: A replica that is cut off must either refuse requests (consistent) or answer with data that may be old (available). No number of replicas removes that choice.
Not true: CAP is one choice, made once for the whole system.
In fact: Different data needs different choices. A bank balance may need consistency, while a count of likes can favour availability, in the same system.
In the interview
It depends on the data. For money or stock levels I'd choose strong consistency, accepting that some requests fail during a partition. For feeds, likes or view counts I'd choose availability with eventual consistency, because a slightly stale number is harmless and the system stays up.
Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.