celery | 易学教程

Celery &Rabbitmq:WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?- a experiment on the GIT

阅读更多关于 Celery &Rabbitmq:WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?- a experiment on the GIT

Recently , I am doing an experiment on a GIT project to understanding the big data processing framework. 1、GIT project： https://github.com/esperdyne/celery-message-processing we have the following components: 1、AMPQ broker( RabbitMQ ): it works as a message buffer, which works as a mail-box to exchange messages for different user! 2、worker： it works as the service-server to provide service for various service client. 3、Queue( "celery" ：it works as a multi-processing container which is used to handle the various worker instances at the same time. the key configuration can be seen as bellow: We

Creating a Celery worker using node.js

阅读更多关于 Creating a Celery worker using node.js

问题 Using node-celery, we can enable node to push Celery jobs to the task queue. How can we allow node to be a Celery worker and consume the queue? 回答1: For Celery if the end point is amqp. Checkout Celery.js Github any node process started as amqp consumer would work fine. For every other self.conf.backend_type types you can have varied consumer. Following example is merely for amqp. One such example. The message below may be the Celery task object. var amqp = require('amqp'); var connection =

Retrying celery failed tasks that are part of a chain

阅读更多关于 Retrying celery failed tasks that are part of a chain

I have a celery chain that runs some tasks. Each of the tasks can fail and be retried. Please see below for a quick example: from celery import task @task(ignore_result=True) def add(x, y, fail=True): try: if fail: raise Exception('Ugly exception.') print '%d + %d = %d' % (x, y, x+y) except Exception as e: raise add.retry(args=(x, y, False), exc=e, countdown=10) @task(ignore_result=True) def mul(x, y): print '%d * %d = %d' % (x, y, x*y) and the chain: from celery.canvas import chain chain(add.si(1, 2), mul.si(3, 4)).apply_async() Running the two tasks (and assuming that nothing fails), your

How can I disable the Django Celery admin modules?

阅读更多关于 How can I disable the Django Celery admin modules?

问题 I have no need to the celery modules in my Django admin. Is there a way I could remove it? 回答1: To be more specific, in admin.py of any app inside INSTALLED_APPS after 'djcelery' from django.contrib import admin from djcelery.models import ( TaskState, WorkerState, PeriodicTask, IntervalSchedule, CrontabSchedule) admin.site.unregister(TaskState) admin.site.unregister(WorkerState) admin.site.unregister(IntervalSchedule) admin.site.unregister(CrontabSchedule) admin.site.unregister(PeriodicTask)

Why does Celery work in Python shell, but not in my Django views? (import problem)

阅读更多关于 Why does Celery work in Python shell, but not in my Django views? (import problem)

问题 I installed Celery (latest stable version.) I have a directory called /home/myuser/fable/jobs . Inside this directory, I have a file called tasks.py: from celery.decorators import task from celery.task import Task class Submitter(Task): def run(self, post, **kwargs): return "Yes, it works!!!!!!" Inside this directory, I also have a file called celeryconfig.py: BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = "abc" BROKER_PASSWORD = "xyz" BROKER_VHOST = "fablemq" CELERY_RESULT

Workaround for celery task priority on RabbitMQ?

阅读更多关于 Workaround for celery task priority on RabbitMQ?

问题 I am running Django with Celery on top of RabbitMQ as a queue to handle some data processing tasks. I am kicking off celery tasks when a user first signs up, as well as periodically to update their data. However, I'd like to of course give priority to the tasks running users who are currently online. I noticed there was a priority setting for tasks in celery, but it seems that rabbitmq does not support this. This thread http://groups.google.com/group/celery-users/browse_thread/thread

Running a task after all tasks have been completed

阅读更多关于 Running a task after all tasks have been completed

问题 I'm writing an application which needs to run a series of tasks in parallel and then a single task with the results of all the tasks run: @celery.task def power(value, expo): return value ** expo @celery.task def amass(values): print str(values) It's a very contrived and oversimplified example, but hopefully the point comes across well. Basically, I have many items which need to run through power , but I only want to run amass on the results from all of the tasks. All of this should happen

Should django model object instances be passed to celery?

阅读更多关于 Should django model object instances be passed to celery?

问题 # models.py from django.db import models class Person(models.Model): first_name = models.CharField(max_length=30) last_name = models.CharField(max_length=30) text_blob = models.CharField(max_length=50000) # tasks.py import celery @celery.task def my_task(person): # example operation: does something to person # needs only a few of the attributes of person # and not the entire bulky record person.first_name = person.first_name.title() person.last_name = person.last_name.title() person.save() In

Why does celery add thousands of queues to rabbitmq that seem to persist long after the tasks completel?

阅读更多关于 Why does celery add thousands of queues to rabbitmq that seem to persist long after the tasks completel?

问题 I am using celery with a rabbitmq backend. It is producing thousands of queues with 0 or 1 items in them in rabbitmq like this: $ sudo rabbitmqctl list_queues Listing queues ... c2e9b4beefc7468ea7c9005009a57e1d 1 1162a89dd72840b19fbe9151c63a4eaa 0 07638a97896744a190f8131c3ba063de 0 b34f8d6d7402408c92c77ff93cdd7cf8 1 f388839917ff4afa9338ef81c28aad75 0 8b898d0c7c7e4be4aa8007b38ccc00ea 1 3fb4be51aaaa4ac097af535301084b01 1 This seems to be inefficient, but further I have observed that these

What is the best option for a (Python 3) task queue on Windows now that Celery 4 has dropped Windows support?

阅读更多关于 What is the best option for a (Python 3) task queue on Windows now that Celery 4 has dropped Windows support?

问题 We run a Flask site under IIS on Windows, and for out-of-process tasks we use Celery. Celery has given us some problems under Windows, but for now we are satisfied running version 3.1.12, using RabbitMQ/AMQP as a back-end, which works under Windows. The new version of Celery (4) has dropped support for Windows, so I'm looking for a viable alternative. RQ seems a very nice task queue, but it also does not support Windows (bottom of the page) I have seen some more, seemingly less popular task