rabbitmq queues filling up with celery tasks

问题

I am using Celery to call multiple hardware units by their ip address. Each unit will return a list of values. Application code below

# create a list of tasks
modbus_calls = []
for site in sites:
    call = call_plc.apply_async((site.name, site.address), expires=120)  # expires after 2 minutes?
    modbus_calls.append(call)

# below checks all tasks are complete (values returned), then move forward out of the while loop 
ready_list = [False]
while not all(ready_list):
    ready_list = []
    for task in modbus_calls:
        ready_list.append(task.ready())

# once here, all tasks have returned their values. use the task.get() method to obtain the list of values

Within the tasks.py file, call_plc task is defined as

@app.task
def call_plc(sitename, ip_address):
    vals = pc.PLC_Comm().connect_to(sitename, ip_address)
    return vals

What is happening: I can only run this application for a certain number of times before rabbitmq starts crashing (running out of memory). I look in /var/lib/rabbitmq/mnesia/rabbit@mymachine/queues and I see a bunch of queues with uuid names. These uuid names do not match the names of the task id's (learned from print task.id within my application). Every time I run the application, there are n queues added to this folder, where n = number of sites to call.

the first time I run the application after resetting rabbitmq, it adds n+1 queues

How can I make it so these tasks / queues do not persist? Once I get the results, I no longer need the task in any way.

task.forget() fails with NotImplementedError('backend does not implement forget.')

The task expiration settings seem to have no effect. My celeryconfig file is below:

BROKER_URL = 'amqp://webdev_rabbit:password@localhost:5672/celeryhost'
CELERY_RESULT_BACKEND = 'amqp://webdev_rabbit:password@localhost:5672/celeryhost'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT=['json']
CELERY_TIMEZONE = 'Europe/Oslo'
CELERY_ENABLE_UTC = True
CELERY_AMQP_TASK_RESULT_EXPIRES = 120

回答1:

It sounds like you don't want to use RabbmitMQ as a result backend, only as a message broker. See this previous question: Queues with random GUID being generated in RabbitMQ server

来源：https://stackoverflow.com/questions/28650684/rabbitmq-queues-filling-up-with-celery-tasks

标签

python

multithreading

RabbitMQ

celery

mnesia