celery

In my /etc/defaults/celeryd config file, I've set: CELERYD_NODES="agent1 agent2 agent3 agent4 agent5 agent6 agent7 agent8" CELERYD_OPTS="--autoscale=10,3 --concurrency=5" I understand that the daemon spawns 8 celery workers, but I'm fully not sure what autoscale and concurrency do together. I thought that concurrency was a way to specify the max number of threads that a worker can use and autoscale was a way for the worker to scale up and down child workers, if necessary. The tasks have a largish payload (some 20-50kB) and there are like 2-3 million such tasks, but each task runs in less than

有业务线提出需求：要求对于其流量，只能在0点到7点扫描。 对此，celery发送任务到队列时可以指定执行的时间。 当worker收到任务后，判断还未到执行时间，会存储在worker中，在到达时候后再执行。 如果还未执行就中断worker，则任务会重新打回celery队列中，不担心丢失。 所以只需要传入time格式的具体执行时间就行。 Demo import datetime def in_run_time(start, end): """ 用来给任务判断，在不在可执行的时间里，是不是需要丢到定时里 Args: start: 任务开始执行的时间，格式如 "00:00:00" end: 任务停止执行的时间，格式如 "07:00:00" Returns: """ current_date = str(datetime.datetime.now().date()) + " " start_time = datetime.datetime.strptime(current_date + start, '%Y-%m-%d %H:%M:%S') end_time = datetime.datetime.strptime(current_date + end, '%Y-%m-%d %H:%M:%S') current_date = datetime.datetime.now() if (start

I have managed to get periodic tasks working in django-celery by subclassing PeriodicTask. I tried to create a test task and set it running doing something useless. It works. Now I can't stop it. I've read the documentation and I cannot find out how to remove the task from the execution queue. I have tried using celeryctl and using the shell, but registry.tasks() is empty, so I can't see how to remove it. I have seen suggestions that I should "revoke" it, but for this I appear to need a task id, and I can't see how I would find the task id. Thanks. A task is a message, and a "periodic task"

I've define a Celery app in a module, and now I want to start the worker from the same module in its __main__ , i.e. by running the module with python -m instead of celery from the command line. I tried this: app = Celery('project', include=['project.tasks']) # do all kind of project-specific configuration # that should occur whenever this module is imported if __name__ == '__main__': # log stuff about the configuration app.start(['worker', '-A', 'project.tasks']) but now Celery thinks I'm running the worker without arguments: Usage: worker <command> [options] Show help screen and exit.

Since nobody provided a solution to this post plus the fact that I desperately need a workaround, here is my situation and some abstract solutions/ideas for debate. My stack: Tornado Celery MongoDB Redis RabbitMQ My problem: Find a way for Tornado to dispatch a celery task ( solved ) and then asynchronously gather the result ( any ideas? ). Scenario 1: (request/response hack plus webhook) Tornado receives a (user)request, then saves in local memory (or in Redis) a { jobID : (user)request} to remember where to propagate the response, and fires a celery task with jobID When celery completes the

How to use Django with AWS Elastic Beanstalk that would also run tasks by celery on main node only? This is how I set up celery with django on elastic beanstalk with scalability working fine. Please keep in mind that 'leader_only' option for container_commands works only on environment rebuild or deployment of the App. If service works long enough, leader node may be removed by Elastic Beanstalk. To deal with that, you may have to apply instance protection for your leader node. Check: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#instance-protection

Is there a way to determine if any task is lost and retry it? I think that the reason for lost can be dispatcher bug or worker thread crash. I was planning to retry them but I'm not sure how to determine which tasks need to be retired? And how to make this process automatically? Can I use my own custom scheduler which will create new tasks? Edit: I found from the documentation that RabbitMQ never loose tasks, but what happens when worker thread crash in the middle of task execution? What you need is to set CELERY_ACKS_LATE = True Late ack means that the task messages will be acknowledged after

I'm trying to keep multiple celery queues with different tasks and workers in the same redis database. Really just a convenience issue of only wanting one redis server rather than two on my machine. I followed the celery tutorial docs verbatim, as it as the only way to get it to work for me. Now when I try to duplicate everything with slightly tweaked names/queues, it keeps erroring out. Note - I'm a newish to Python and Celery, which is obviously part of the problem. I'm not sure which parts are named "task/tasks" as a name vs special words. My condensed version of docs: Run celery -A tasks