How to use airflow with Celery

早过忘川 提交于 2019-12-04 02:43:47

The Airflow documentation covers this quite nicely:

First, you will need a celery backend. This can be for example Redis or RabbitMQ. Then, the executor parameter in your airflow.cfg should be set to CeleryExecutor.

Then, in the celery section of the airflow.cfg, set the broker_url to point to your celery backend (e.g. redis://your_redis_host:your_redis_port/1). Point celery_result_backend to a sql database (you can use the same as your main airflow db).

Then, on your worker machines simply kick off airflow worker and your jobs should start on the two machines.

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!