Scalability: vertical vs horizontal

Scalability is a system's ability to handle more load by adding resources.

Why it matters

A system that works for 100 users can fail at 100,000. You need a way to add capacity without rewriting everything.

The idea

Vertical scaling means a bigger machine (more CPU and memory). It's simple, but there's a ceiling, and that one machine can fail. Horizontal scaling means more machines behind a load balancer. It has no hard ceiling and survives failures, but needs stateless app servers and shared storage. The key trick: keep app servers stateless, and keep sessions and data in shared stores.

A busy restaurant can buy a bigger oven (vertical) or open more kitchens (horizontal).

More kitchens only help if any kitchen can cook any order. If half of a customer's order is in kitchen 2, it has to go back there. That's 'state', and it's why app servers should be stateless.

How it works

  1. Find the bottleneck first: is the CPU at 100%, the database slow, memory full, or the network saturated?
  2. Make app servers stateless: no sessions or files kept on the server; put them in Redis and object storage instead.
  3. Put a load balancer in front and add servers. With auto-scaling, servers are added automatically when CPU use or latency rises.
  4. Take reads off the database: cache hot data and add read replicas.
  5. When writes become the bottleneck, shard the database by a key.
  6. Move slow work off the request path with queues.

Example

Moving login sessions from server memory into Redis lets any server handle any user, so you can add servers freely.

Key points

Trade-offs

Common mistakes

Not true: A slow system: more app servers make it faster, even when the database is the bottleneck.
In fact: Only when the app servers are the bottleneck. If the database is the slow part, more app servers just send it more work, so measure first.

Not true: Sessions in a server's memory: they keep working when more servers are added.
In fact: The next request may reach a different server, which knows nothing about the session. Keep sessions in a shared store, so the app servers stay stateless.

In the interview

I'd keep the application tier stateless so it scales horizontally behind a load balancer with auto-scaling. State goes into shared stores: sessions in Redis, data in the database. For the database, I'd add read replicas first and shard only when write volume needs it.

Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.