Load balancing

A load balancer spreads incoming requests across several servers and stops sending traffic to unhealthy ones.

Why it matters

One server can only handle so much, and if it dies, everyone using it is affected. You need to spread requests across many servers and route around the broken ones.

The idea

It makes many servers look like one, enables horizontal scaling and hides failures. Layer 4 balancers route by IP address and port (fast). Layer 7 balancers read the HTTP request, so they can route by path, header or cookie. Common algorithms: round robin, least connections, and hashing (the same user always goes to the same server).

A supermarket worker sending shoppers to the shortest checkout queue, and closing a till when it breaks.

Shoppers are fairly similar. Requests aren't: one may take 5 ms and another 5 seconds, so 'the same number of requests' isn't 'the same amount of work'. Least connections helps.

How it works

  1. Clients connect to the load balancer's address, never directly to a server.
  2. For each request or connection, it picks a healthy server using an algorithm: the next in turn (round robin), or the one with the fewest active connections.
  3. Every few seconds it calls each server's health endpoint. After a few failures in a row it stops sending traffic there, and adds the server back when it recovers.
  4. A layer 7 balancer can read the URL and send /api to API servers and /images to image servers.
  5. So the balancer itself isn't a single point of failure, run two of them (one active, one standby, sharing an address) or use a managed service.

Example

An AWS Application Load Balancer (layer 7) sends /api/* to API servers and /images/* to a media service, and checks each server's /health endpoint every few seconds.

Key points

Common mistakes

Not true: A health check that finds the process running proves the server can handle requests.
In fact: A process can be running and still be unable to reach its database. A good health check tests what a real request needs.

Not true: Round robin sends each request to the server with the fewest open connections.
In fact: Round robin simply takes the servers in turn. Sending each request to the server with the fewest open connections is the least-connections algorithm.

In the interview

I'd put a layer 7 load balancer in front of the stateless app servers, with health checks and least-connections routing. It runs as a redundant pair or a managed service, so it isn't a single point of failure.

Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.