Unit 3: Data: databases, storage and scaling data
Choose the right database, keep data correct with transactions, make queries fast with indexes, and scale data with replication and sharding.
The big picture
App servers are easy to scale; data is not. Most deep dives in interviews end up here: which store, which key, how many copies, how to split it, and what consistency you promise.
Lessons in this unit
- SQL vs NoSQL: choosing a database: SQL (relational) databases store data in tables with a fixed schema and support joins and transactions. NoSQL is a family of other models: key-value, document, wide-column and graph.
- ACID transactions and isolation levels: A transaction groups several operations so they succeed or fail together. ACID stands for Atomic, Consistent, Isolated, Durable.
- Database indexing: An index is an extra data structure that lets the database find rows without scanning the whole table.
- Replication: Replication keeps copies of the same data on several machines.
- Partitioning and sharding: Sharding splits a large dataset across several databases (shards), each holding part of the data.
- Consistent hashing: Consistent hashing assigns keys to servers so that adding or removing a server moves only a small share of the keys.
- Object storage (S3) for files and media: Object storage keeps files ('objects') such as images, videos and backups as a key pointing to bytes plus metadata, in a cheap, very durable, almost unlimited store like Amazon S3.
- Search: inverted indexes and Elasticsearch: Full-text search finds the documents that contain some words and ranks them by relevance. It's built on an inverted index: a map from each word to the documents containing it.
- Unique ID generation: Generating IDs that are unique across many servers, without one shared database counter.