Asynchronous processing and message queues (Kafka, SQS)

Asynchronous processing means doing slow work later, outside the user's request. A message queue holds the work, or events, between the service that produces them and the services that consume them.

Why it matters

Some work is slow (encoding video, sending emails) or comes in bursts (a sale doubles traffic). Doing it inside the user's request makes them wait, and a burst can overwhelm the services behind it.

The idea

Instead of making a user wait while you send emails, resize images and update analytics, you put a message on a queue and reply straight away. Workers process messages at their own pace. Queues absorb spikes, decouple services and let you retry failures. SQS-style queues give each message to one worker; Kafka keeps an ordered log that many groups of consumers can read, and re-read.

A restaurant's order rail. The waiter clips the order on the rail and goes back to the customers; the cooks take orders when they're ready.

A paper order is taken by exactly one cook and then it's gone. Most queues only promise 'at least once' delivery, so a message can arrive twice and consumers must be idempotent. And Kafka keeps messages after they're read, so others can read them again.

How it works

  1. A user uploads a video. The API stores the file and puts a message {video_id: 42} on a queue.
  2. The API replies straight away: '202 Accepted, processing'.
  3. Worker processes pull messages from the queue. One worker takes message 42 and converts the video.
  4. When it's done, the worker acknowledges the message, and only then is it removed. If the worker crashes first, the message becomes visible again and another worker retries it.
  5. If it fails too many times, it moves to a dead-letter queue for a person to inspect.
  6. If 10,000 uploads arrive at once, they simply wait in the queue; you add workers to clear it faster.

Example

Upload a video: the API saves the file, puts a 'video uploaded' message on a queue, and replies 202 Accepted. Workers convert it into several resolutions, and the user is notified when it's ready.

Key points

Trade-offs

Common mistakes

Not true: Delivery: a queue hands each message to its consumer exactly once.
In fact: Most queues deliver at least once: after a crash or a timeout, a message can arrive twice. Make processing idempotent, so that a repeat does no harm.

Not true: Kafka topic: messages stay in order across the whole topic.
In fact: Order is kept only within one partition. Messages that must stay in order need the same key, so that they go to the same partition.

In the interview

Slow or non-critical work goes on a queue so the request stays fast. For simple task distribution I'd use a queue like SQS with a dead-letter queue. For event streams that many services need, or that I may want to replay, I'd use Kafka, partitioned by a key like user ID to keep each user's events in order. Consumers are idempotent, because delivery is at least once.

Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.