celery

How can I detect whether I'm running in a Celery worker?

由 為{幸葍}努か 提交于 2019-12-01 17:40:06
问题 Is there a way to determine, programatically, that the current module being imported/run is done so in the context of a celery worker? We've settled on setting an environment variable before running the Celery worker, and checking this environment variable in the code, but I wonder if there's a better way? 回答1: Simple, import sys IN_CELERY_WORKER_PROCESS = sys.argv and sys.argv[0].endswith('celery')\ and 'worker' in sys.argv if IN_CELERY_WORKER_PROCESS: print ('Im in Celery worker') http:/

Airflow: Tasks queued but not running

由 泄露秘密 提交于 2019-12-01 17:25:18
I am new to airflow and trying to setup airflow to run ETL pipelines. I was able to install airflow postgres celery rabbitmq I am able to test run the turtorial dag. When i try to schedule the jobs, scheduler is able to pick it up and queue the jobs which i could see on the UI but tasks are not running. Could somebody help me fix ths issue? I believe i am missing most basic airflow concept here. below is the airflow.cfg Here is my config file: [core] airflow_home = /root/airflow dags_folder = /root/airflow/dags base_log_folder = /root/airflow/logs executor = CeleryExecutor sql_alchemy_conn =

Celery dies with DBPageNotFoundError

由 懵懂的女人 提交于 2019-12-01 17:16:32
问题 I have 3 machines with celery workers and rabbitmq as a broker, one worker is running with beat flag, all of this is managed by supervisor, and sometimes celery dies with such error. This error appears only on beat worker, but when it appears, workers on all machines dies. (celery==3.1.12, kombu==3.0.20) [2014-07-05 08:37:04,297: INFO/MainProcess] Connected to amqp://user:**@192.168.15.106:5672// [2014-07-05 08:37:04,311: ERROR/Beat] Process Beat Traceback (most recent call last): File "/var

Celery dies with DBPageNotFoundError

由 孤街醉人 提交于 2019-12-01 17:04:27
I have 3 machines with celery workers and rabbitmq as a broker, one worker is running with beat flag, all of this is managed by supervisor, and sometimes celery dies with such error. This error appears only on beat worker, but when it appears, workers on all machines dies. (celery==3.1.12, kombu==3.0.20) [2014-07-05 08:37:04,297: INFO/MainProcess] Connected to amqp://user:**@192.168.15.106:5672// [2014-07-05 08:37:04,311: ERROR/Beat] Process Beat Traceback (most recent call last): File "/var/projects/env/local/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap self.run()

Celery: ignoring BROKER_URL in config file

由 China☆狼群 提交于 2019-12-01 17:03:50
My question is a duplicate of this one , but more detailed. The problem is that I have a BROKER_URL set in my Celery config file, but that isn't reflected in and I am loading the config: I checked, and it is being loaded - in fact, other constants defined there are being set, just not BROKER_URL . This appears to be a bug, but I wanted to be sure. celeryconfig.py : BROKER_URL = "amqp://user:password@remote.server.com:5672//vhost" CELERY_TASK_SERIALIZER = 'json' CELERY_RESULT_SERIALIZER = 'json' CELERY_ENABLE_UTC = True ( JSON is being used as the serializer, not Pickle , so I know this is

工作日志-20180707

由 旧城冷巷雨未停 提交于 2019-12-01 16:30:51
摘要 「HHBchain」中使用了 CCXT 而且目前的项目是使用 celery 做的分布式，多个 CCXT 对象会在多个 celery worker 进程中被重复使用，这样就会存在 nonce 被重复的问题，所以这里必须对 CCXT 对象加锁。目前在研究锁方案。 「django-easy-validator」今天修复了几个小 BUG， 基本上已经能够使用了，最近要发一个新版本呢。 统计 工作时段 : 9:00 - 12:00, 14:00 - 19:00 工作时长 : 8 Hours 代码 : 4 Commits 读书 : 1 Hours 写作 : 0.1 Hour 练琴 : 1 Hour 发呆 大专栏 工作日志-20180707 : 0 Hour 备注 关于锁的实现我想到了两种方案，一种是使用 python 自带的锁，每一种 CCXT 对象生成一个锁，各自锁各自的，这样在只有一个 celery worker 的时候是起作用的，因为在 fork 之前各个进程已经获取到锁了，但是在分布式的情况下就完全不起作用了。 原因是每个 celery woker 都会有自己的环境变量，所以锁没有共享，也就无法加锁， 这个时候只能使用 linux 提供的文件锁来解决这个问题，一个 CCXT exchange 对象一个文件，这样就可以解决在分布式环境下的锁问题了。 这种方案还有待测试

Airflow celery worker will be blocked if sensor number large than concurrency?

由 可紊 提交于 2019-12-01 16:25:05
Let's say, I set celery concurrency to n , but I have m ( m > n ) ExternalTaskSensor in a dag, it will check another dag named do_sth , these ExternalTaskSensor will consume all celery worker, so that no one will work in fact. But I can't set concurreny too high(like 2* m ), because dag do_sth may start too many process which will lead to out of memory. I am confused what number I should set to celery concurrency? In ETL best practices with Airflow's Gotchas section the author addresses this general problem. One of the suggestions is to setup a pool for your sensor tasks so that your other

Airflow celery worker will be blocked if sensor number large than concurrency?

由 半世苍凉 提交于 2019-12-01 16:09:19
问题 Let's say, I set celery concurrency to n , but I have m ( m > n ) ExternalTaskSensor in a dag, it will check another dag named do_sth , these ExternalTaskSensor will consume all celery worker, so that no one will work in fact. But I can't set concurreny too high(like 2* m ), because dag do_sth may start too many process which will lead to out of memory. I am confused what number I should set to celery concurrency? 回答1: In ETL best practices with Airflow's Gotchas section the author addresses

Celery not picking CELERY_ALWAYS_EAGER settings

由 我的梦境 提交于 2019-12-01 15:37:32
I am running Django 1.8 + Celery 4.0.2 Celery is configured well and can run my Django tasks locally on redis backend. But when I try to use CELERY_ALWAYS_EAGER = True settings, this settings has no effect. Which is not the case with other settings, e.g. CELERY_TIMEZONE Specifically, in pdb I see that app.conf.task_always_eager is False lib/python2.7/site-packages/celery/app/task.py(520)apply_async() So somehow CELERY_ALWAYS_EAGER is not picked up and has no effect on app.conf.task_always_eager More info from pdb: > app.conf.get('CELERY_ALWAYS_EAGER') > True > app.conf.task_always_eager >

Celery not picking CELERY_ALWAYS_EAGER settings

由 你。 提交于 2019-12-01 14:35:59
问题 I am running Django 1.8 + Celery 4.0.2 Celery is configured well and can run my Django tasks locally on redis backend. But when I try to use CELERY_ALWAYS_EAGER = True settings, this settings has no effect. Which is not the case with other settings, e.g. CELERY_TIMEZONE Specifically, in pdb I see that app.conf.task_always_eager is False lib/python2.7/site-packages/celery/app/task.py(520)apply_async() So somehow CELERY_ALWAYS_EAGER is not picked up and has no effect on app.conf.task_always
订阅 celery