celery

I am running Django with Celery on top of RabbitMQ as a queue to handle some data processing tasks. I am kicking off celery tasks when a user first signs up, as well as periodically to update their data. However, I'd like to of course give priority to the tasks running users who are currently online. I noticed there was a priority setting for tasks in celery, but it seems that rabbitmq does not support this. This thread http://groups.google.com/group/celery-users/browse_thread/thread/ac3b6123d63421e5/b7740def1389e87e?lnk=gst&q=priority#b7740def1389e87e suggests have two different queues, a

I installed Celery (latest stable version.) I have a directory called /home/myuser/fable/jobs . Inside this directory, I have a file called tasks.py: from celery.decorators import task from celery.task import Task class Submitter(Task): def run(self, post, **kwargs): return "Yes, it works!!!!!!" Inside this directory, I also have a file called celeryconfig.py: BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = "abc" BROKER_PASSWORD = "xyz" BROKER_VHOST = "fablemq" CELERY_RESULT_BACKEND = "amqp" CELERY_IMPORTS = ("tasks", ) In my /etc/profile , I have these set as my PYTHONPATH:

I'm writing an application which needs to run a series of tasks in parallel and then a single task with the results of all the tasks run: @celery.task def power(value, expo): return value ** expo @celery.task def amass(values): print str(values) It's a very contrived and oversimplified example, but hopefully the point comes across well. Basically, I have many items which need to run through power , but I only want to run amass on the results from all of the tasks. All of this should happen asynchronously, and I don't need anything back from the amass method. Does anyone know how to set this up

# models.py from django.db import models class Person(models.Model): first_name = models.CharField(max_length=30) last_name = models.CharField(max_length=30) text_blob = models.CharField(max_length=50000) # tasks.py import celery @celery.task def my_task(person): # example operation: does something to person # needs only a few of the attributes of person # and not the entire bulky record person.first_name = person.first_name.title() person.last_name = person.last_name.title() person.save() In my application somewhere I have something like: from models import Person from tasks import my_task

I'm using Django Celery task to connect to Facebook Graph API with requests lib using Gevent. Issue I'm constantly running at is that every now and then I get EOF occurred in violation of protocol exception. I've searched around and various sources offer different fixes but none seems to work. I've tried monkey patching the ssl module(gevent.monkey.patch_all()) and some others too but no luck. I'm not even sure if this is openssl issue as some sources might suggest as I haven't encountered it before applying Gevent optimisation Connection error: [Errno 8] _ssl.c:504: EOF occurred in violation

　　我的环境： celery 3.1.25 python 3.6.9 window10 celery tasks 代码如下，其中 QuotesSpider 是我的scrapy项目爬虫类名称 from celery_app import appfrom scrapy.crawler import CrawlerProcessfrom scrapy.utils.project import get_project_settingsfrom tutorial.spiders.quotes import QuotesSpider def crawl_run(): scope = 'all' process = CrawlerProcess(settings=get_project_settings()) process.crawl(QuotesSpider, scope) process.start() process.join()@app.task(queue='default')def execute_task(): return crawl_run() 来源： https://www.cnblogs.com/WalkOnMars/p/11558560.html

I do retries with celery like in the Docs-Example: @task() def add(x, y): try: ... except Exception, exc: add.retry(exc=exc, countdown=60) # override the default and # retry in 1 minute How can I increase the retry-countdown everytime the retry occurs for this job - e.g. 60 seconds, 2 minutes, 4 minutes and so on until the MaxRetriesExceeded is raised? Since version 4.2 you can use options autoretry_for and retry_backoff for this purposes, for example: @task(max_retries=10, autoretry_for=(Exception,), retry_backoff=60) def add(x, y): pass Here is a simple way to create bigger delay each time