Dynamic dags not getting added by scheduler

Question

I am trying to create Dynamic DAGs and then get them to the scheduler. I tried the reference from https://www.astronomer.io/guides/dynamically-generating-dags/ which works w

Answer 1

I guess doing the following would fix it

• completely remove the global testDynDags dag and tst_dyn_dags task (instantiation and invocation)
• invoke your fetch_new_dags(..) method with requisite arguments in global scope

---

Explanation

• Dynamic dags / tasks merely means that you have a well-defined logic at the time of writing dag-definition file that can help create tasks / dags having a known structure in a pre-defined fashion.
• You can NOT determine the structure of your DAG at runtime (task execution). So, for instance, you cannot add n identical tasks to your DAG if the upstream task returned an integer value n. But you can iterate over a YAML file containing n segments and generate n tasks / dags.

So clearly, wrapping dag-generation code inside an Airflow task itself makes no sense.

---

UPDATE-1

From what is indicated in comments, I infer that the requirement dictates that you revise your external source that feeds inputs (how many dags or tasks to create) to your DAG / task-generation script. While this is indeed a complex use-case, but a simple way to achieve this is to create 2 separate DAGs.

• One dag runs once in a while and generates the inputs that are stored in an an external resource like Airflow Variable (or any other external store like file / S3 / database etc.)
• The second DAG is constructed programmatically by reading that same datasource which was written by the first DAG

You can take inspiration from the Adding DAGs based on Variable value section