airflow-scheduler

Airflow 1.9.0 is queuing but not launching tasks

不羁岁月 提交于 2019-11-26 04:24:55
问题 Airflow is randomly not running queued tasks some tasks dont even get queued status. I keep seeing below in the scheduler logs [2018-02-28 02:24:58,780] {jobs.py:1077} INFO - No tasks to consider for execution. I do see tasks in database that either have no status or queued status but they never get started. The airflow setup is running https://github.com/puckel/docker-airflow on ECS with Redis. There are 4 scheduler threads and 4 Celery worker tasks. For the tasks that are not running are