how to clear failing DAGs using the CLI in airflow

问题

I have some failing DAGs, let's say from 1st-Feb to 20th-Feb. From that date upword, all of them succeeded.

I tried to use the cli (instead of doing it twenty times with the Web UI):

airflow clear -f -t * my_dags.my_dag_id

But I have a weird error:

airflow: error: unrecognized arguments: airflow-webserver.pid airflow.cfg airflow_variables.json my_dags.my_dag_id

EDIT 1:

Like @tobi6 explained it, the * was indeed causing troubles. Knowing that, I tried this command instead:

airflow clear -u -d -f -t ".*" my_dags.my_dag_id

but it's only returning failed task instances (-f flag). -d and -u flags don't seem to work because taskinstances downstream and upstream the failed ones are ignored (not returned).

EDIT 2:

like @tobi6 suggested, using -s and -e permits to select all DAG runs within a date range. Here is the command:

airflow clear  -s "2018-04-01 00:00:00" -e "2018-04-01 00:00:00"  my_dags.my_dag_id.

However, adding -f flag to the command above only returns failed task instances. is it possible to select all failed task instances of all failed DAG runs within a date range ?

回答1:

If you are using an asterik * in the Linux bash, it will automatically expand the content of the directory.

Meaning it will replace the asterik with all files in the current working directory and then execute your command.

This will help to avoid the automatic expansion:

"airflow clear -f -t * my_dags.my_dag_id"

回答2:

One solution I've found so far is by executing sql(MySQL in my case):

update task_instance t left join dag_run d on d.dag_id = t.dag_id and d.execution_date = t.execution_date
set t.state=null,
    d.state='running'
where t.dag_id = '<your_dag_id'
  and t.execution_date > '2020-08-07 23:00:00'
  and d.state='failed';

It will clear all tasks states on failed dag_runs, as button 'clear' pressed for entire dag run in web UI.

来源：https://stackoverflow.com/questions/50130975/how-to-clear-failing-dags-using-the-cli-in-airflow

标签

airflow