Rate limiting and backpressure

Rate limiting caps how many requests a client can make in a time window. Backpressure is a system telling the senders upstream to slow down when it's overloaded.

Why it matters

Without limits, one buggy script, scraper or attacker can send so many requests that everyone else gets errors. And a system that accepts more work than it can do eventually collapses for everyone.

The idea

Rate limits protect services from abuse, bugs and traffic spikes, and enforce fair use, like 100 requests per minute per user. Requests over the limit get HTTP 429. Common algorithms: token bucket, leaky bucket, fixed window and sliding window.

Backpressure goes further: queues have size limits, and when they fill, senders are told to wait or requests are dropped on purpose, instead of the whole system falling over.

A nightclub bouncer lets in a few people per minute (rate limiting). When the club is full, the queue outside stops moving (backpressure).

A bouncer is one person at one door. A real service runs on many servers, so the limit must be shared between them, usually with counters in Redis, and that shared store must be fast and always available.

How it works

  1. Choose the rule: for example, 10 requests per second per API key, with bursts of up to 20.
  2. Token bucket: each key has a bucket that holds up to 20 tokens and gains 10 tokens every second.
  3. Each request takes one token. If a token is there, the request goes through.
  4. If the bucket is empty, reply 429 Too Many Requests, with 'Retry-After: 1'.
  5. With many gateway servers, keep each bucket in Redis and update it atomically, so every server shares the same count.
  6. Inside the system, backpressure: when a work queue reaches its limit, stop accepting new work (or drop the least important) instead of letting the queue grow forever.

Example

Token bucket: each user's bucket holds up to 10 tokens and refills at 1 token per second. Each request takes a token; an empty bucket means 429 Too Many Requests. Short bursts are allowed; sustained abuse isn't.

Key points

Common mistakes

Not true: Counters kept on each server: the limit still holds across the whole system.
In fact: With ten servers, a client could get ten times the limit. Keep the counters in a shared store such as Redis, so that every server sees the same count.

Not true: One global limit shared by the users: a fair design, since no one gets special treatment.
In fact: With one shared limit, a single heavy user can use it all up and block everyone else. Limit each user, API key or IP address separately.

In the interview

I'd rate-limit at the API gateway with a token bucket per user or API key, keeping the counters in Redis so every gateway node shares them, and return 429 with Retry-After. Internally I'd use bounded queues and load shedding, so overload slows the system down gracefully instead of crashing it.

Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.