Asynchronous processing and message queues (Kafka, SQS)
Asynchronous processing means doing slow work later, outside the user's request. A message queue holds the work, or events, between the service that produces them and the services that consume them.
Why it matters
Some work is slow (encoding video, sending emails) or comes in bursts (a sale doubles traffic). Doing it inside the user's request makes them wait, and a burst can overwhelm the services behind it.
The idea
Instead of making a user wait while you send emails, resize images and update analytics, you put a message on a queue and reply straight away. Workers process messages at their own pace. Queues absorb spikes, decouple services and let you retry failures. SQS-style queues give each message to one worker; Kafka keeps an ordered log that many groups of consumers can read, and re-read.
A restaurant's order rail. The waiter clips the order on the rail and goes back to the customers; the cooks take orders when they're ready.
A paper order is taken by exactly one cook and then it's gone. Most queues only promise 'at least once' delivery, so a message can arrive twice and consumers must be idempotent. And Kafka keeps messages after they're read, so others can read them again.
How it works
- A user uploads a video. The API stores the file and puts a message {video_id: 42} on a queue.
- The API replies straight away: '202 Accepted, processing'.
- Worker processes pull messages from the queue. One worker takes message 42 and converts the video.
- When it's done, the worker acknowledges the message, and only then is it removed. If the worker crashes first, the message becomes visible again and another worker retries it.
- If it fails too many times, it moves to a dead-letter queue for a person to inspect.
- If 10,000 uploads arrive at once, they simply wait in the queue; you add workers to clear it faster.
Example
Upload a video: the API saves the file, puts a 'video uploaded' message on a queue, and replies 202 Accepted. Workers convert it into several resolutions, and the user is notified when it's ready.
Key points
- Queue (SQS, RabbitMQ): spreads work; each message is consumed once, then deleted.
- Log or stream (Kafka): ordered, durable, replayable; partitions give order per key and parallel processing.
- Delivery guarantees: at most once, at least once (the most common), exactly once (hard; usually 'effectively once' with idempotent consumers).
- Dead-letter queue: where messages go after failing too many times.
- Pub/sub: one event fans out to many subscribers.
- Async means eventual: the user sees 'processing', not the final result straight away.
Trade-offs
- Pros: absorbs spikes, decouples services, makes retries easy.
- Cons: more moving parts, harder debugging, eventual consistency, and you must handle ordering and duplicates.
Common mistakes
- Assuming each message arrives exactly once. It can arrive twice, so make processing idempotent.
- Expecting order across a whole Kafka topic. Order is only kept within one partition.
- Using a queue where the user needs an immediate answer.
Not true: Delivery: a queue hands each message to its consumer exactly once.
In fact: Most queues deliver at least once: after a crash or a timeout, a message can arrive twice. Make processing idempotent, so that a repeat does no harm.
Not true: Kafka topic: messages stay in order across the whole topic.
In fact: Order is kept only within one partition. Messages that must stay in order need the same key, so that they go to the same partition.
In the interview
Slow or non-critical work goes on a queue so the request stays fast. For simple task distribution I'd use a queue like SQS with a dead-letter queue. For event streams that many services need, or that I may want to replay, I'd use Kafka, partitioned by a key like user ID to keep each user's events in order. Consumers are idempotent, because delivery is at least once.
Part of Unit 2: Core concepts: scale, reliability and speed in the free System Design course.