Get all Airflow Leaf Nodes/Tasks

前端 未结 1 1563
無奈伤痛
無奈伤痛 2020-12-11 18:04

I want to build something where I need to capture all of the leaf tasks and add a downstream dependency to them to make a job complete in our database. Is there an easy way

相关标签:
1条回答
  • 2020-12-11 18:18

    Use upstream_task_ids and downstream_task_ids @property from BaseOperator

    def get_start_tasks(dag: DAG) -> List[BaseOperator]:
        # returns list of "head" / "root" tasks of DAG
        return [task for task in dag.tasks if not task.upstream_task_ids]
    
    
    def get_end_tasks(dag: DAG) -> List[BaseOperator]:
        # returns list of "leaf" tasks of DAG
        return [task for task in dag.tasks if not task.downstream_task_ids]
    

    Type-Annotations from Python 3.6+


    UPDATE-1

    Now Airflow DAG model has powerful @property functions like

    • leaves
    • roots
    • topological_sort
    0 讨论(0)
提交回复
热议问题