Load balancing
A load balancer spreads incoming requests across several servers and stops sending traffic to unhealthy ones.
Why it matters
One server can only handle so much, and if it dies, everyone using it is affected. You need to spread requests across many servers and route around the broken ones.
The idea
It makes many servers look like one, enables horizontal scaling and hides failures. Layer 4 balancers route by IP address and port (fast). Layer 7 balancers read the HTTP request, so they can route by path, header or cookie. Common algorithms: round robin, least connections, and hashing (the same user always goes to the same server).
A supermarket worker sending shoppers to the shortest checkout queue, and closing a till when it breaks.
Shoppers are fairly similar. Requests aren't: one may take 5 ms and another 5 seconds, so 'the same number of requests' isn't 'the same amount of work'. Least connections helps.
How it works
- Clients connect to the load balancer's address, never directly to a server.
- For each request or connection, it picks a healthy server using an algorithm: the next in turn (round robin), or the one with the fewest active connections.
- Every few seconds it calls each server's health endpoint. After a few failures in a row it stops sending traffic there, and adds the server back when it recovers.
- A layer 7 balancer can read the URL and send /api to API servers and /images to image servers.
- So the balancer itself isn't a single point of failure, run two of them (one active, one standby, sharing an address) or use a managed service.
Example
An AWS Application Load Balancer (layer 7) sends /api/* to API servers and /images/* to a media service, and checks each server's /health endpoint every few seconds.
Key points
- Algorithms: round robin, weighted round robin, least connections, IP or consistent hashing.
- Health checks remove failed servers automatically.
- Layer 4: fast and protocol-agnostic. Layer 7: smarter routing and TLS handling, but more work per request.
- Sticky sessions tie a user to one server; avoid them by keeping servers stateless.
- The load balancer itself must be redundant (a pair, or a managed service).
- Load balancers are also used between internal services, not only at the edge.
Common mistakes
- Sticky sessions that tie one user to a server forever, which breaks when that server dies.
- Health checks that only test 'is the process running?', not 'can it reach its database?'.
- Forgetting that the load balancer needs redundancy too.
Not true: A health check that finds the process running proves the server can handle requests.
In fact: A process can be running and still be unable to reach its database. A good health check tests what a real request needs.
Not true: Round robin sends each request to the server with the fewest open connections.
In fact: Round robin simply takes the servers in turn. Sending each request to the server with the fewest open connections is the least-connections algorithm.
In the interview
I'd put a layer 7 load balancer in front of the stateless app servers, with health checks and least-connections routing. It runs as a redundant pair or a managed service, so it isn't a single point of failure.
Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.