Message Queue Patterns

Message queues are foundational components in distributed systems, enabling asynchronous communication, decoupling services, and absorbing traffic spikes.

Mental Model

By separating message production from consumption, queues allow applications to remain responsive and scale independently. Producers publish messages to a central Broker. In point-to-point (Lists), messages are consumed by a single competing worker via destructive pop (BRPOP). In publish-subscribe (Pub/Sub), messages are fanned out in real-time to all active subscribers; this channel is ephemeral, so offline subscribers do not receive old messages unless a durable stream is used. In streaming logs (Streams), messages are appended to a persistent log and can be read by Consumer Groups, where each message is delivered to a single group consumer. Streams delivery records a pending reference in the Pending Entries List (PEL) while the stream entry remains, until explicitly acknowledged with XACK (which removes the handled message from the PEL to prevent reclaims). Because Redis Streams has no native partitions, ordered IDs are sequenced chronologically in a single log, but concurrent consumer processing and completion can occur out of order as XREADGROUP ... > delivers newer entries while earlier ones remain pending in the PEL. Persistently failing poison messages are quarantined via an application-managed policy: after reaching 3 failed delivery attempts (observed via PEL delivery counts), the worker explicitly writes to XADD stream:orders:dlq and clears the original PEL entry with XACK. Because at-least-once delivery can redeliver unacknowledged messages after crashes or timeouts, consumers must be designed to safely handle potential duplicate processing.

Components and Why They Exist

Producers

The client applications that generate and send event payloads. Serializes message data and publishes it to the broker via LPUSH, PUBLISH, or XADD. In the demo, watch request envelopes flow from producers to the broker.

Message Broker (Redis)

The central messaging middleware managing message routing and buffers. Depending on the pattern selected, stores messages in lists (LPUSH/BRPOP), channels (PUBLISH), or append-only streams (XADD), managing bounded internals like queue buffers, stream logs, and the Pending Entries List (PEL). In the demo, watch it route, store, and manage message lists or logs.

Consumers

The worker applications that retrieve and process message records. Polls, receives copies, or reads assigned stream entries via consumer groups, executing the core processing logic for each message. In the demo, watch them poll, receive broadcast copies, or consume stream records concurrently.

Consumer Groups

Enables parallel message processing of a single stream across multiple worker nodes. Divides stream consumption among registered group consumers without native partitions. Monotonically ordered stream IDs are delivered via XREADGROUP ... >, which can deliver newer entries while earlier entries remain pending in the PEL, meaning consumer processing and completion can occur out of order. Consumer-group at-least-once delivery can cause duplicate processing during worker failures, requiring idempotent consumers. In the demo, watch messages distribute among group consumers.

Pending List (PEL) / Acknowledgment (ACK)

Tracks in-flight messages delivered to consumers within a group. Streams delivery records a pending reference in the PEL while the stream entry remains. XACK merely removes a successfully handled message from the PEL so it is not reclaimed by peers. If a message is not acknowledged, it remains in the PEL and can be claimed via XCLAIM/XAUTOCLAIM for redelivery. In the demo, watch unacknowledged messages wait in the pending list.

Dead Letter Queue (DLQ)

Isolates poison messages that cause repeated consumer processing failures. Redis Streams does not natively count retries or perform automated DLQ routing. Instead, an application-managed policy inspects PEL delivery counters (via XPENDING or XAUTOCLAIM/XCLAIM). When a message reaches 3 failed delivery attempts, the consumer application explicitly writes the malformed payload to a dead letter stream using XADD stream:orders:dlq and issues XACK on the source stream to delete the stalled entry from the PEL, enabling offline troubleshooting without blocking the queue. In the demo, watch failed messages get redirected to the DLQ node.

Edge Cases and Misconceptions

• Misconception: Redis Streams have native topic partitions like Kafka: Redis Streams have no native partitions. A stream is a single, append-only chronological log of monotonically increasing IDs (timestamp-sequence). Consumer groups divide message distribution among workers reading from that single log. Because workers process concurrently and at varying speeds, message completion can occur out of order, and XREADGROUP ... > delivers newer unread entries even while older entries remain unacknowledged in the PEL.
• Misconception: Redis Streams automatically manages retry limits and dead letter routing: Redis Streams does not automatically count retries or send failed messages to a Dead Letter Queue (DLQ). The broker only maintains the delivery counter inside the Pending Entries List (PEL) metadata. Enforcing a retry threshold (such as 3 failed delivery attempts), redirecting poison messages to a secondary stream via XADD stream:orders:dlq, and removing the failed message from the primary stream’s PEL via XACK is entirely an application-managed policy implemented in worker code.
• Misconception: Destructive list pop guarantees reliable delivery: Redis Lists (LPUSH/BRPOP) remove the message immediately upon retrieval. If the consumer crashes before processing completes, the message is lost permanently, creating an at-most-once loss risk. Reliable at-least-once delivery requires Redis Streams with consumer groups and explicit acknowledgment (XACK).
• Misconception: XACK guarantees delivery: XACK does not guarantee delivery; it only confirms that a message was successfully processed and tells Redis to remove it from the Pending Entries List (PEL). If a worker crashes before sending XACK, the message remains in the PEL and will be reclaimed for redelivery, which can cause duplicate processing.
• Misconception: Consumer groups prevent duplicate processing: Consumer groups deliver each new message to one consumer under normal conditions, but failures or timeouts trigger reclaims from the PEL, meaning at-least-once delivery can cause duplicate processing if a consumer fails after processing but before XACK.
• Misconception: Pub/Sub is a durable queuing mechanism: Redis Pub/Sub is fire-and-forget. If a subscriber is offline when a message is published, the message is permanently lost. Durability requires a persistent queue (like Redis Lists or Redis Streams).
• Misconception: Point-to-point queues automatically fan-out messages: Point-to-point (using LPUSH/BRPOP) operates on a competing consumers model where only one consumer grabs and processes a message. Fan-out behavior is achieved using Pub/Sub or Streams with separate consumer groups.
• Edge Case: Unacknowledged Message Leaks (OOM): In Redis Streams, delivered messages record pending references in the Pending Entries List (PEL) while stream entries remain. Until they are explicitly acknowledged with XACK, the PEL entries persist. If consumers crash or fail to acknowledge processed messages, the PEL will grow indefinitely, consuming Redis memory and potentially causing an Out-Of-Memory (OOM) crash.