When does a airflow dag definition get evaluated?

不问归期 提交于 2020-01-30 09:16:25

问题


Suppose I have an airflow dag file that creates a graph like so...

def get_current_info(filename)
    current_info = {}
    <fill in info in current_info relevant for today's date for given file>
    return current_info

files = [
    get_current_info("file_001"),
    get_current_info("file_002"),
    ....
]

for f in files:
    <some BashOperator bo1 using f's current info dict>
    <some BashOperator bo2 using f's current info dict>
    ....

    bo1 >> bo2
    ....

Since these values in the current_info dict that is used to define the dag changes periodically (here, daily), I would like to know by what process / schedule the dag definition gets updated. (I print the current_info values each run and values appear to be updating, but curious as to how and when exactly this happens).

When does a airflow dag definition get evaluated? referenced anywhere in the docs?


回答1:


The DAGs are evaluated in every run of the scheduler.

This article describes how the scheduler works and at what stage the DAG files are picked up for evaluation.



来源:https://stackoverflow.com/questions/58567386/when-does-a-airflow-dag-definition-get-evaluated

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!