celery

I use celery to update RSS feeds in my news aggregation site. I use one @task for each feed, and things seem to work nicely. There's a detail that I'm not sure to handle well though: all feeds are updated once every minute with a @periodic_task, but what if a feed is still updating from the last periodic task when a new one is started ? (for example if the feed is really slow, or offline and the task is held in a retry loop) Currently I store tasks results and check their status like this: import socket from datetime import timedelta from celery.decorators import task, periodic_task from

How can I retrieve a list of tasks in a queue that are yet to be processed? semarj EDIT: See other answers for getting a list of tasks in the queue. You should look here: Celery Guide - Inspecting Workers Basically this: >>> from celery.task.control import inspect # Inspect all nodes. >>> i = inspect() # Show the items that have an ETA or are scheduled for later processing >>> i.scheduled() # Show tasks that are currently active. >>> i.active() # Show tasks that have been claimed by workers >>> i.reserved() Depending on what you want if you are using rabbitMQ, use this in terminal: sudo

I have a Django site where a scrape happens when a user requests it, and my code kicks off a Scrapy spider standalone script in a new process. Naturally, this isn't working with an increase of users. Something like this: class StandAloneSpider(Spider): #a regular spider settings.overrides['LOG_ENABLED'] = True #more settings can be changed... crawler = CrawlerProcess( settings ) crawler.install() crawler.configure() spider = StandAloneSpider() crawler.crawl( spider ) crawler.start() I've decided to use Celery and use workers to queue up the crawl requests. However, I'm running into issues with

I have been reading the doc and searching but cannot seem to find a straight answer: Can you cancel an already executing task? (as in the task has started, takes a while, and half way through it needs to be cancelled) I found this from the doc at Celery FAQ >>> result = add.apply_async(args=[2, 2], countdown=120) >>> result.revoke() But I am unclear if this will cancel queued tasks or if it will kill a running process on a worker. Thanks for any light you can shed! mher revoke cancels the task execution. If a task is revoked, the workers ignore the task and do not execute it. If you don't use

I just found out about the configuration option CELERYD_PREFETCH_MULTIPLIER ( docs ). The default is 4, but (I believe) I want the prefetching off or as low as possible. I set it to 1 now, which is close enough to what I'm looking for, but there's still some things I don't understand: Why is this prefetching a good idea? I don't really see a reason for it, unless there's a lot of latency between the message queue and the workers (in my case, they are currently running on the same host and at worst might eventually run on different hosts in the same data center). The documentation only mentions

This is not working anymore , scrapy's API has changed. Now the documentation feature a way to " Run Scrapy from a script " but I get the ReactorNotRestartable error. My task: from celery import Task from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy import log, signals from scrapy.utils.project import get_project_settings from .spiders import MySpider class MyTask(Task): def run(self, *args, **kwargs): spider = MySpider settings = get_project_settings() crawler = Crawler(settings) crawler.signals.connect(reactor.stop, signal=signals.spider_closed) crawler