Airflow force re-run of upstream task when cleared even though downstream if marked success

China☆狼群 提交于 2019-12-10 23:50:40

问题


I have tasks A -> B -> C in Airflow and when I run the DAG and all complete with success, I'd like to be able to clear B alone (while leaving C marked as success). B clears and gets put into the 'no_status' state but then when I try to re-run B, nothing happens. I've tried --ignore_dependencies, --ignore_depends_on_past and --force but to no avail. B seems to only re-run if C is also cleared and then everything re-runs as expected.

The reason why I'd like to be able re-run B specifically without changing the pipeline is that some of B's external inputs may change slightly (file changed, or tweak) and I'd like to run it and evaluate it's output before restarting the downstream tasks (to mitigate any potential interruption).


回答1:


If you clear via the command line, you can specify which tasks you want cleared.

airflow clear your_dag_id --task_regex B --start_date 2017-09-25T17:00:00 --end_date 2017-09-25T17:00:00

If you want to do this via the UI, you can go to Browse -> Task Instances to find the task instance you want to clear. Deleting from there should only clear the tasks you selected. You will also need to go over to Browse -> DAG Run and set the relevant DAG run to the running state for your cleared task to be picked up.




回答2:


In the UI, when you clear a instance task, the downstream case is checked by default.

If you unchecked it, it will clear only this one and not re-run the downstream tasks



来源:https://stackoverflow.com/questions/46397373/airflow-force-re-run-of-upstream-task-when-cleared-even-though-downstream-if-mar

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!