Airflow dynamic DAG and Task Ids

后端 未结 2 1896
猫巷女王i
猫巷女王i 2020-12-09 03:32

I mostly see Airflow being used for ETL/Bid data related jobs. I\'m trying to use it for business workflows wherein a user action triggers a set of dependent tasks in future

2条回答
  •  星月不相逢
    2020-12-09 04:20

    From How can I create DAGs dynamically?:

    Airflow looks in you [sic] DAGS_FOLDER for modules that contain DAG objects in their global namespace, and adds the objects it finds in the DagBag. Knowing this all we need is a way to dynamically assign variable in the global namespace, which is easily done in python using the globals() function for the standard library which behaves like a simple dictionary.

    for i in range(10):
        dag_id = 'foo_{}'.format(i)
        globals()[dag_id] = DAG(dag_id)
        # or better, call a function that returns a DAG object!
    

提交回复
热议问题