Creating dynamic tasks in airflow (in composer) based on bigquery response

问题

I am trying to create a airflow DAG which generates task depending on the response from server.

Here is my approach :

getlist of tables from bigquery -> loop through the list and create tasks

This is my latest code and I have tried all possible code found in stack overflow. Nothing seems to work. What am I doing wrong?

with models.DAG(dag_id="xt", default_args=default_args, schedule_interval="0 1 * * *", catchup=True) as dag:
tables = get_tables_from_bq()

    bridge = DummyOperator(
        task_id='bridge',
        dag=dag
    )


    for t in tables:
        sql = ("SELECT * FROM `{project}.{dataset}.{table}` LIMIT 5;".format(
                project=project, dataset=dataset, table=t))

    materialize_t = BigQueryOperator(bql=sql,
                                     destination_dataset_table=dataset+'.' + table_prefix + t,
                                     task_id = 'x_' + t,
                                     bigquery_conn_id = 'bigquery_default',
                                     use_legacy_sql = False,
                                     write_disposition = 'WRITE_APPEND',
                                     create_disposition = 'CREATE_IF_NEEDED',
                                     query_params = {},
                                     allow_large_results = True,
                                     dag = dag)

bridge >> materialize_t

Even the run option is not showing with this code. I tried multiple codes and finally reached here but still no luck. Any help???

回答1:

I don't know if it is a typo in the copy and paste of the DAG but tables = get_tables_from_bq() should be before with models.DAG(...) Also, bridge >> materialize_t seems to miss indentation and therefore be outside the with models.DAG(...) scope. On a side note, you do not need the bridge task.

来源：https://stackoverflow.com/questions/57689837/creating-dynamic-tasks-in-airflow-in-composer-based-on-bigquery-response