Using Celery with existing RabbitMQ messages

Question

I have an existing RabbitMQ deployment that that a few Java applications are using the send out log messages as string JSON objects on various channels. I would like to use Celery to consume these messages and write them to various places (e.g. DB, Hadoop, etc.).

I can see that Celery is design to be both the producer and consumer of RabbitMQ messages, since it tries to hide the mechanism by which those messages are delivered. Is there anyway to get Celery to consume messages created by another app and run jobs when they arrive?

Answer 1

It's currently hard to add custom consumers to the celery workers, but this is changing in the development version (to become 3.1) where I've added support for Consumer boot-steps.

There's no documentation yet as I've just finished implementing it, but here's an example:

from celery import Celery
from celery.bin import Option
from celery.bootsteps import ConsumerStep
from kombu import Consumer, Exchange, Queue

class CustomConsumer(ConsumerStep):
   queue = Queue('custom', Exchange('custom'), routing_key='custom')

   def __init__(self, c, enable_custom_consumer=False, **kwargs):
       self.enable = self.enable_custom_consumer

   def get_consumers(self, connection):
       return [
           Consumer(connection.channel(),
               queues=[self.queue],
               callbacks=[self.on_message]),
       ]

   def on_message(self, body, message):
       print('GOT MESSAGE: %r' % (body, ))
       message.ack()


celery = Celery(broker='amqp://localhost//')
celery.steps['consumer'].add(CustomConsumer)
celery.user_options['worker'].add(
    Option('--enable-custom-consumer', action='store_true',
           help='Enable our custom consumer.'),
)

Note that the API may change in the final version, one thing that I'm not yet sure about is how channels are handled after get_consumer(connection). Currently the channel of the consumer is closed when connection is lost, and at shutdown, but people may want to handle channels manually. In that case there's always the possibility of customizing ConsumerStep, or writing a new StartStopStep.