Best practice to configure your RabbitMq cluster hosted with GCP's Kubernetes engine

This article discusses few best practices that we have used with our RabbitMq cluster hosted with GCP's Kubernetes application. Note that every configuration comes with a certain trade-off. It might work for us but not necessarily for your different requirement and workload.

Photo by T S on Unsplash

Set lazy mode for queues

With this mode it basically means that the messages delivered through RabbitMq are persisted to disk, instead of being kept in memory. This comes at a cost of higher latency in our case, however it brings much more stability for us.

This setting helps us to avoid the case when suddenly the number of messages spike and it needs more than the limit set by us for the RabbitMq's pods. As a result, we do not need to worry if the RabbitMq's pods might crash in those cases.

To set this mode using policy to all queues, you can run this command:

Set time-to-live (TTL) for messages

Unprocessed messages can stay for a long time in the queue (even for forever) and prevent new tasks from being processed. Thus, we need to configure a maximum time that a message can live in the queue.

Use dead-letter queue to store unprocessed messages

When a message is failed to be processed by the consumer or it exceeds the time-to-live duration, it should not be discarded by RabbitMq but rather should be kept in a separate dead-letter queue so we can investigate the message's content and why it failed later.

Here we can configure a dead-letter exchange with RabbitMq so a message can be sent there and in turn, being forwarded to the dead letter queue.

It's also configure different dead-letter queue for different topics using routing key.

Set max length for dead-letter queue

Since all failed messages can be sent to the dead-letter queue, we have the risk that the queue itself can grow indefinitely. Thus it makes sense to only keep a certain amount of failed messages.

How is your experience in configuring the RabbitMq cluster? Feel free to share your thoughts here through the comment section.

Product engineer / novice writer (alexthered.me)