How to find the number of upstream tasks failed in Airflow?

问题

I am having a tough time in figuring out how to find the failed task for the same dag run running twice on same day(same execution day).

Consider an example when a dag with dag_id=1 has failed on the first run (due to any reason lets say connection timeout maybe) and task got failed. TaskInstance table will contain the entry of the failed task when we try to query it. GREAT!!

But, If I re-run the same dag(note that dag_id is still 1) then in the last task(it has the rule of ALL_DONE so irrespective of the whether upstream task was failed or was successful it will be executed) I want to calculate the number of tasks failed in the current dag_run ignoring the previous dag_runs. I came across dag_run id which could be useful if we can relate it to TaskInstance but I could not. Any suggestions/help is appreciated.

回答1:

You could create a PythonOperator task which queries the Airflow database to find the information you're looking for. This has the added benefit of passing along the information you need to query for the data you want:

from contextlib import closing
from airflow import models, settings
from airflow.utils.state import State

def your_python_operator_callable(**context):    
  with closing(settings.Session()) as session:
    print("There are {} failed tasks in this execution".format(
      session.query(
        models.TaskInstance
      ).filter(
        models.TaskInstance.dag_id == context["dag"].dag_id, 
        models.TaskInstance.execution_date == context["execution_date"],
        models.TaskInstance.state == State.FAILED).count()
      )

Then add the task to your DAG with a PythonOperator.

(I have not tested the above, but hopefully will send you on the right path)

回答2:

In Airflow 1.10.x the same result can be achieved by much simpler code that avoids touching ORM directly.

from airflow.utils.state import State

def your_python_operator_callable(**context):    
    tis_dagrun = context['ti'].get_dagrun().get_task_instances()
    failed_count = sum([True if ti.state == State.FAILED else False for ti in tis_dagrun])
    print(f"There are {failed_count} failed tasks in this execution"

The one unfortunate problem is that context['ti'].get_dagrun(), when testing in CLI a single task, does not return instance of DAGRun so you have to go around that.

来源：https://stackoverflow.com/questions/50613155/how-to-find-the-number-of-upstream-tasks-failed-in-airflow

标签

airflow