Message Queues and Pub/Sub
Put a durable buffer between services so they can work at different speeds, and learn how Kafka partitions, offsets and delivery guarantees behave.
What a message queue is
A message queue is a durable buffer between two services. A producer (the sender) writes a message (a small piece of data describing work or an event). A consumer (the receiver) reads it later and processes it.
Without a queue, service A calls service B directly. If B is slow or down, A is slow or down too. With a queue in between:
- Decoupling. A does not know who consumes, or how many consumers there are.
- Spike absorption. A burst of 10,000 messages waits in the queue instead of crashing the consumer.
- Async work. The user gets a reply now; the email, thumbnail or report is produced afterwards.
- Retry. A failed message can be tried again without the producer knowing.
A broker is the server that stores and delivers messages. Common ones: RabbitMQ, Amazon SQS (Simple Queue Service), Apache Kafka, Google Pub/Sub.
1 of 8