airflow

I have already seen this and this questions on SO and made the changes accordingly. However, my dependent DAG still gets stuck in poking state. Below is my master DAG: from airflow import DAG from airflow.operators.jdbc_operator import JdbcOperator from datetime import datetime from airflow.operators.bash_operator import BashOperator today = datetime.today() default_args = { 'depends_on_past': False, 'retries': 0, 'start_date': datetime(today.year, today.month, today.day), 'schedule_interval': '@once' } dag = DAG('call-procedure-and-bash', default_args=default_args) call_procedure =

I'm using a clustered Airflow environment where I have four AWS ec2-instances for the servers. ec2-instances Server 1: Webserver, Scheduler, Redis Queue, PostgreSQL Database Server 2: Webserver Server 3: Worker Server 4: Worker My setup has been working perfectly fine for three months now but sporadically about once a week I get a Broken Pipe Exception when Airflow is attempting to log something. *** Log file isn't local. *** Fetching here: http://ip-1-2-3-4:8793/log/foobar/task_1/2018-07-13T00:00:00/1.log [2018-07-16 00:00:15,521] {cli.py:374} INFO - Running on host ip-1-2-3-4 [2018-07-16 00

In Airflow, I'd like a job to run at specific time each day in a non-UTC timezone. How can I go about scheduling this? The problem is that once daylight savings time is triggered, my job will either be running an hour too soon or an hour too late. In the Airflow docs , it seems like this is a known issue: In case you set a cron schedule, Airflow assumes you will always want to run at the exact same time. It will then ignore day light savings time. Thus, if you have a schedule that says run at end of interval every day at 08:00 GMT+1 it will always run end of interval 08:00 GMT+1, regardless if

I'm still in the process of deploying Airflow and I've already felt the need to merge operator s together. The most common use-case would be coupling an operator and the corresponding sensor . For instance, one might want to chain together the EmrStepOperator and EmrStepSensor . I'm creating my DAG s programmatically , and the biggest one of those contains 150+ (identical) branches , each performing the same series of operations on different bits of data (tables). Therefore clubbing together tasks that make-up a single logical step in my DAG would be of great help. Here are 2 contending

I just started using apache airflow. I am trying to run test.sh file from airflow, however it is not work. Following is my code, file name is test.py import os from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime, timedelta default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2015, 6, 1), 'email': ['airflow@airflow.com'], 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), # 'queue': 'bash_queue', # 'pool': 'backfill', # 'priority_weight': 10, # 'end

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_for_product1) ├ -> (do_work_for_product2) ├ -> (do_work_for_product3) ├ … So the start task has multiple downstreams. And I setup concurrency related configuration as below: parallelism = 3 dag_concurrency = 3 max_active_runs = 1 Then when I run this DAG manually (not sure if it never happens on a scheduled task) , some downstreams get executed, but others stuck at "queued" status. If I clear the task from