celery-task

I am looking for some advice as to the best way to map a list generated from a task to another task in celery. Let's say I have a task called parse , which parses a PDF document and outputs a list of pages. Each page then needs to be individually passed to another task called feed . This all needs to go inside a task called process So, one way I could do that is this: @celery.task def process: pages = parse.s(path_to_pdf).get() feed.map(pages) Of course, that is not a good idea because I am calling get() inside a task. Additionally this is inefficient, since my parse task is wrapped around a

I am starting a set of celery tasks by using celery group as described in the official documentation I am also storing the group (taskset) id into a db, in order to poll celery for the taskset state. job = group([ single_test.s(1, 1), single_test.s(1, 2), single_test.s(1, 3), ]) result = job.apply_async() test_set = MyTestSet() test_set.taskset_id = result.id # store test_set into DB Is there a way to obtain a GroupResult object (i.e. my result ) starting from the taskset id? Something like what is done in this question , but working with celery groups. I already tried doing: r = GroupResult

I installed django-celery and I tried to start up the worker server but I get an OSError that a function isn't implemented. I'm running CentOS release 5.4 (Final) on a VPS: . broker -> amqp://guest@localhost:5672/ . queues -> . celery -> exchange:celery (direct) binding:celery . concurrency -> 4 . loader -> djcelery.loaders.DjangoLoader . logfile -> [stderr]@WARNING . events -> OFF . beat -> OFF [2010-07-22 17:10:01,364: WARNING/MainProcess] Traceback (most recent call last): [2010-07-22 17:10:01,364: WARNING/MainProcess] File "manage.py", line 11, in <module> [2010-07-22 17:10:01,364: WARNING

I've a python celery-redis queue processing uploads and downloads worth gigs and gigs of data at a time. Few of the uploads takes upto few hours. However once such a task finishes, I'm witnessing this bizarre celery behaviour that the celery scheduler is rerunning the just concluded task again by sending it again to the worker (I'm running a single worker) And it just happened 2times on the same task! Can someone help me know why is this happening and how can I prevent it? The tasks are definitely finishing cleanly with no errors reported just that these are extremely long running tasks. I

I have a celery chain that runs some tasks. Each of the tasks can fail and be retried. Please see below for a quick example: from celery import task @task(ignore_result=True) def add(x, y, fail=True): try: if fail: raise Exception('Ugly exception.') print '%d + %d = %d' % (x, y, x+y) except Exception as e: raise add.retry(args=(x, y, False), exc=e, countdown=10) @task(ignore_result=True) def mul(x, y): print '%d * %d = %d' % (x, y, x*y) and the chain: from celery.canvas import chain chain(add.si(1, 2), mul.si(3, 4)).apply_async() Running the two tasks (and assuming that nothing fails), your

How can I delete all pending tasks without knowing the task_id for each task? Philip Southam From the docs : $ celery -A proj purge or from proj.celery import app app.control.purge() (EDIT: Updated with current method.) ToonAlfrink For celery 3.0+: $ celery purge To purge a specific queue: $ celery -Q queue_name purge smido For Celery 2.x and 3.x: When using worker with -Q parameter to define queues, for example celery worker -Q queue1,queue2,queue3 then celery purge will not work, because you cannot pass the queue params to it. It will only delete the default queue. The solution is to start