Airflow : ExternalTaskSensor doesn't trigger the task

Question

I have already seen this and this questions on SO and made the changes accordingly. However, my dependent DAG still gets stuck in poking state. Below is my master DAG:

Answer 1

Make sure both DAGs start at the same time and you don't start either DAGs manually.

Answer 2

Hope you are not triggering DAG manually. If you want to test it let the DAG run as per the schedule and then monitor the DAG runs.

Answer 3

I had this problem because of a summer/winter time change: "1 day before" means "exactly 24 hours before" so if the time zone has daylight savings time change in between, the DAG is stuck.

One way out of this is to manually set it as successful.

The other way would be to use the execution_date_fn argument and manually calculate the time difference correctly in this case.

Answer 4

It may be that you should use a positive timedelta: https://airflow.readthedocs.io/en/stable/_modules/airflow/sensors/external_task_sensor.html because when subtracting the execution delta it's going to end up looking for a task that ran 2 minutes after itself.

However the delta isn't really a range, the TI has to have a matching Dag ID, Task ID, successful result and also an execution date in the list of datetimes. Which when you give execution_delta as a delta, is a list of one datetime taking the current execution date and subtracting the timedelta.

This probably comes down to you either removing the timedelta so that the two execution dates match and the sensor will wait until the other task is successful, OR your start date and schedule interval being set as basically today and @once are getting execution dates not in predictable lock-step with each other. You could try setting say datetime(2019,1,10) and 0 1 * * * to get them to both run daily at 1am (again without an execution_delta).