celery

Celery beat schedule, schedule to run on load then on interval

别来无恙 提交于 2021-01-27 03:53:19
问题 I am trying to figure out how to configure a periodic task in celery to be scheduled to run on load regardless of interval. For example, beat_schedule = { 'my-task': { 'task': 'module.my_task', 'schedule': 60.0, }, } will wait 60 seconds after the beat is started to run for the first time. This is problematic for a longer interval, such as an hour, that can do work that is immediately valuable but is not needed "fresh" at shorter intervals. This question addresses this issue but neither of

Celery beat schedule, schedule to run on load then on interval

允我心安 提交于 2021-01-27 03:52:14
问题 I am trying to figure out how to configure a periodic task in celery to be scheduled to run on load regardless of interval. For example, beat_schedule = { 'my-task': { 'task': 'module.my_task', 'schedule': 60.0, }, } will wait 60 seconds after the beat is started to run for the first time. This is problematic for a longer interval, such as an hour, that can do work that is immediately valuable but is not needed "fresh" at shorter intervals. This question addresses this issue but neither of

What is the default Celery log level if none is specified?

こ雲淡風輕ζ 提交于 2021-01-27 03:51:17
问题 According to the Celery documentation, the -l/--loglevel command line option can be used for: -l, --loglevel Logging level, choose between DEBUG, INFO, WARNING, ERROR, CRITICAL, or FATAL. If this option is not used and therefore no log level is specified, what is the log level used by Celery by default? 回答1: Default log level in Celery is WARNING due implementation here. That level is set during initialization of the Logging instance. 来源: https://stackoverflow.com/questions/44428064/what-is

python启动celery进行运行环境检测

有些话、适合烂在心里 提交于 2021-01-13 14:37:57
celery 5.0后启动需要使用 celery 命令行启动服务,比如我们启动服务需要使用 celery -A task.celery_task worker --loglevel=info --concurrency=1 我需要在程序启动前检查是否能正常运行的需求,需要启动前增加检查运行环境是否正常。 celery 命令其实就是一个python文件,在bin目录下有一个celery.py文件, import re import sys from celery.__main__ import main from pywkmisc import HttpClientUtils, get_config # 这个是自己写的包 https://gitee.com/lovelong1/pywktools # python celeryd.py -A task.celery_task worker --loglevel=info --concurrency=1 def check_url(url, fname): try: HttpClientUtils.request_head(url) except Exception as e: print('{fname}无法访问接口{url},系统退出'.format(url=url,fname=fname)) import sys sys.exit

#celery#集群管理实现

扶醉桌前 提交于 2021-01-12 20:27:36
本来这个方案打算用在我的Sora上,但是因为某些问题打算弃用celery。但既然有人想问怎样实现多机器的管理,那就写出来了: 架构: 这里作为例子的celery app为myapp: root@workgroup0:~/celeryapp# ls myapp agent.py celery.py config.py __init__.py root@workgroup0:~/celeryapp# 公用代码部分: celery.py:(备注:172.16.77.175是任务发布节点的ip地址) from __future__ import absolute_import from celery import Celery app = Celery('myapp', broker='amqp://guest@172.16.77.175//', backend='amqp://guest@172.16.77.175//', include=['myapp.agent']) app.config_from_object('myapp.config') if __name__ == '__main__': app.start() config.py: from __future__ import absolute_import from kombu import Queue,Exchange

How to safely restart Airflow and kill a long-running task?

一个人想着一个人 提交于 2021-01-07 06:21:49
问题 I have Airflow is running in Kubernetes using the CeleryExecutor. Airflow submits and monitors Spark jobs using the DatabricksOperator. My streaming Spark jobs have a very long runtime (they run forever unless they fail or are cancelled). When pods for Airflow worker are killed while a streaming job is running, the following happens: Associated task becomes a zombie (running state, but no process with heartbeat) Task is marked as failed when Airflow reaps zombies Spark streaming job continues

How to safely restart Airflow and kill a long-running task?

风流意气都作罢 提交于 2021-01-07 06:20:15
问题 I have Airflow is running in Kubernetes using the CeleryExecutor. Airflow submits and monitors Spark jobs using the DatabricksOperator. My streaming Spark jobs have a very long runtime (they run forever unless they fail or are cancelled). When pods for Airflow worker are killed while a streaming job is running, the following happens: Associated task becomes a zombie (running state, but no process with heartbeat) Task is marked as failed when Airflow reaps zombies Spark streaming job continues

How to safely restart Airflow and kill a long-running task?

删除回忆录丶 提交于 2021-01-07 06:18:58
问题 I have Airflow is running in Kubernetes using the CeleryExecutor. Airflow submits and monitors Spark jobs using the DatabricksOperator. My streaming Spark jobs have a very long runtime (they run forever unless they fail or are cancelled). When pods for Airflow worker are killed while a streaming job is running, the following happens: Associated task becomes a zombie (running state, but no process with heartbeat) Task is marked as failed when Airflow reaps zombies Spark streaming job continues

How to safely restart Airflow and kill a long-running task?

给你一囗甜甜゛ 提交于 2021-01-07 06:18:53
问题 I have Airflow is running in Kubernetes using the CeleryExecutor. Airflow submits and monitors Spark jobs using the DatabricksOperator. My streaming Spark jobs have a very long runtime (they run forever unless they fail or are cancelled). When pods for Airflow worker are killed while a streaming job is running, the following happens: Associated task becomes a zombie (running state, but no process with heartbeat) Task is marked as failed when Airflow reaps zombies Spark streaming job continues

How to register Celery task to specific worker?

一笑奈何 提交于 2021-01-04 07:19:22
问题 I am developing web application in Python/Django, and I have several tasks which are running in celery. I have to run task A one at a time so I have created worker with --concurrency=1 and routed task A to that worker using following command. celery -A proj worker -Q A -c 1 -l INFO Everything is working fine as this worker handle task A and other tasks are routed to default queue. But, above worker return all task when I use inspect command to get registered task for worker. That is