airflow

问题 If I am using a BigQueryOperator with a SQL Template, how could I pass an argument to the SQL? File : .sql/query.sql SELECT * FROM `dataset.{{ task_instance.variable_for_execution }} File : dag.py BigQueryOperator( task_id='compare_tables', sql='./sql/query.sql', use_legacy_sql=False, dag=dag, ) 回答1: You can pass an argument in params parameter which can be used in the templated field as follows: BigQueryOperator( task_id='', sql='SELECT * FROM `dataset.{{ params.param1 }}', params={ 'param1'

问题 I am running a Airflow DAG which moves data from GCS to BQ using operator GoogleCloudStorageToBigQueryOperator i am on Airflow version 1.10.2. This task moves data from MySql to BQ(Table partitioned), all this time we were partitioned by Ingestion-time and the incremental load for past three days were working fine when the data was loaded using Airflow DAG. Now we changed the partitioned type to be Date or timestamp on a DATE column from the table, after which we have started getting this

问题 I am new to airflow and I am trying something simple with GoogleCloudStorageDownloadOperator: default_args = { 'start_date': airflow.utils.dates.days_ago(0), 'schedule_interval': None, 'retries': 1, 'retry_delay': timedelta(minutes=5), 'params': { 'work_dir': '/tmp' } } dag = DAG( 'foo', default_args=default_args, description='This is foobar', schedule_interval=timedelta(weeks=1), dagrun_timeout=timedelta(minutes=60)) mock_download = GoogleCloudStorageDownloadOperator( task_id='download-foo

问题 I have a dag which checks for new workflows to be generated (Dynamic DAG) at a regular interval and if found, creates them. (Ref: Dynamic dags not getting added by scheduler ) The above DAG is working and the dynamic DAGs are getting created and listed in the web-server. Two issues here: When clicking on the DAG in web url, it says "DAG seems to be missing" The listed DAGs are not listed using "airflow list_dags" command Error: DAG "app01_user" seems to be missing. The same is for all other

问题 I am trying to fetch results from BigQueryOperator using airflow but I could not find a way to do it. I tried calling the next() method in the bq_cursor member (available in 1.10) however it returns None . This is how I tried to do it import datetime import logging from airflow import models from airflow.contrib.operators import bigquery_operator from airflow.operators import python_operator yesterday = datetime.datetime.combine( datetime.datetime.today() - datetime.timedelta(1), datetime

问题 I am trying to fetch results from BigQueryOperator using airflow but I could not find a way to do it. I tried calling the next() method in the bq_cursor member (available in 1.10) however it returns None . This is how I tried to do it import datetime import logging from airflow import models from airflow.contrib.operators import bigquery_operator from airflow.operators import python_operator yesterday = datetime.datetime.combine( datetime.datetime.today() - datetime.timedelta(1), datetime

问题 I have a simple Airflow DAG which has only one task - stream_from_twitter_to_kafka Here is the code for the DAG: default_args = { "owner": "me", "depends_on_past": False, "start_date": datetime(2020, 1, 20), "email": ["makalaaneesh18@mail.com"], "email_on_failure": False, "email_on_retry": False, "retries": 0, "retry_delay": timedelta(minutes=1), } NO_OF_TWEETS_TO_STREAM = 100 with DAG("stream_from_twitter", catchup=False, default_args=default_args, schedule_interval="@hourly") as dag: task1

问题 Trying to split out Airflow processes onto 2 servers. Server A, which has been already running in standalone mode with everything on it, has the DAGs and I'd like to set it as the worker in the new setup with an additional server. Server B is the new server which would host the metadata database on MySQL. Can I have Server A run LocalExecutor, or would I have to use CeleryExecutor? Would airflow scheduler has to run on the server that has the DAGs right? Or does it have to run on every server

问题 Trying to split out Airflow processes onto 2 servers. Server A, which has been already running in standalone mode with everything on it, has the DAGs and I'd like to set it as the worker in the new setup with an additional server. Server B is the new server which would host the metadata database on MySQL. Can I have Server A run LocalExecutor, or would I have to use CeleryExecutor? Would airflow scheduler has to run on the server that has the DAGs right? Or does it have to run on every server

问题 Trying to split out Airflow processes onto 2 servers. Server A, which has been already running in standalone mode with everything on it, has the DAGs and I'd like to set it as the worker in the new setup with an additional server. Server B is the new server which would host the metadata database on MySQL. Can I have Server A run LocalExecutor, or would I have to use CeleryExecutor? Would airflow scheduler has to run on the server that has the DAGs right? Or does it have to run on every server