Redis Vs RabbitMQ as a data broker/messaging system in between Logstash and elasticsearch

Question

We are defining an architecture to collect log information by Logstash shippers which are installed in various machines and index the data in one elasticsearch server centra

Answer 1

I have been doing some research on this topic. If performance is important and persistence is not, RabbitMQ is a perfect choice. Redis is a technology developed with a different intent.

Following is a list of pros for using RabbitMQ over Redis:

• RabbitMQ uses Advanced Message Queuing Protocol (AMQP) which can be configured to use SSL, additional layer of security.
• RabbitMQ takes approximately 75% of the time Redis takes in accepting messages.
• RabbitMQ supports priorities for messages, which can be used by workers to consume high priority messages first.
• There is no chance of loosing the message if any worker crashes after consuming the message, which is not the case with Redis.
• RabbitMQ has a good routing system to direct messages to different queues.

A few cons for using RabbitMQ:

• RabbitMQ might be a little hard to maintain, hard to debug crashes.
• node-name or node-ip fluctuations can cause data loss, but if managed well, durable messages can solve the problem.