celery

问题 Is it possible to run a chord callback even if the main tasks failed? I've created a chord which I added a bunch of tasks and registered a callback to it. My problem, is that if one of the tasks fail the callback is not triggered, but I would like the callback to be triggered either way. I've tried to register the callback with si() (immutability) callback = tasks.run_delete_rule.si([timestamp]) header = [tasks.run_update_rule.s(i, timestamp) for i in item_ids] result = chord(header)(callback

问题 I have a django application leveraging celery for asynchronous tasks. I've been running into an issue where I reach the max number of redis connections. I am fairly new to both celery and redis. I'm confused because in my config I define - CELERY_REDIS_MAX_CONNECTIONS = 20 which is the limit on my redis plan. For experimentation, I bumped the plan up and that solved the issue. I am confused, however, that I am running into this problem again after defining the max number of connections. I

问题 I'm currently leveraging celery for periodic tasks. I am new to celery. I have two workers running two different queues. One for slow background jobs and one for jobs user's queue up in the application. I am monitoring my tasks on datadog because it's an easy way to confirm my workers a running appropriately. What I want to do is after each task completes, record which queue the task was completed on. @after_task_publish.connect() def on_task_publish(sender=None, headers=None, body=None, *

问题 I'm currently leveraging celery for periodic tasks. I am new to celery. I have two workers running two different queues. One for slow background jobs and one for jobs user's queue up in the application. I am monitoring my tasks on datadog because it's an easy way to confirm my workers a running appropriately. What I want to do is after each task completes, record which queue the task was completed on. @after_task_publish.connect() def on_task_publish(sender=None, headers=None, body=None, *

问题 I am using celery and redis as two services in my docker setup. Configuration is as below: redis: image: redis:latest hostname: redis ports: - "0.0.0.0:6379:6379" command: --requirepass PASSWORD celeryworker: <<: *django depends_on: - redis - postgres command: "celery -E -A rhombus.taskapp worker --beat --scheduler redbeat.schedulers:RedBeatScheduler --loglevel INFO --uid taskmaster --concurrency=5" When I try to build my containers and schedule some jobs once the workers are ready I get an

问题 Below code snippet is the celery setup I have, Here I am running a twisted reactor in each child celery worker process. import os from threading import Thread from celery import Celery from twisted.internet import threads, reactor from celery import signals from twisted.internet.defer import inlineCallbacks, returnValue from twisted.internet.task import LoopingCall app = Celery('tasks', broker='pyamqp://guest@localhost//') @signals.worker_process_init.connect def configure_infrastructure(*

问题 I am trying to run a function in parallel for multiple files and want all of them to terminate before a point. For Example: There is a loop def main(): for item in list: function_x(item) function_y(list) Now what I want is that this function_x should run in parallel for all items. But this function should be executed for all items before my function_y is called. I am planning to use celery for this. but can not understand how to do this. 回答1: Here is my final test code. All I needed to do is

问题 I have a django FileField, which i use to store wav files on the Amazon s3 server. I have set up the celery task to read that file and convert it to mp3 and store it to another FileField. Problem i am facing is that i am unable to pass the input file to ffmpeg as the file is not the physical file on the hard disk drive. To circumvent that, i used stdin to feed the input stream of the file with the django's filefield. Here is the example: output_file = NamedTemporaryFile(suffix='.mp3')

问题 I am trying to make a call to a external Google Cloud Natural Language API from within a Celery task (using the google-cloud-python package). The problem is that the call to the API never returns (hangs): @celery.task() def get_entities_async(): return get_entities() def get_entities(): gcloud_client = LanguageServiceClient() doc = types.Document(content='This is a test.', language='en', type='PLAIN_TEXT') res = gcloud_client.analyze_entities(document=doc) # This call never returns print(

问题 We are doing background data processing with Django Celery, taking a CSV file (up to 15MB), converting it into list of dict data (which also includes some Django model objects), and breaking it up into chunks to process in sub tasks: @task def main_task(data): i = 0 for chunk in chunk_up(data): chunk_id = "chunk_id_{}".format(i) cache.set(chunk_id, chunk, timeout=FIVE_HOURS) sub_task.delay(chunk_id) i += 1 @task def sub_task(chunk_id): data_chunk = cache.get(chunk_id) ... # do processing All