airflow

Is there any way to make a user-defined macro in Airflow which is itself computed from other macros? from airflow import DAG from airflow.operators.bash_operator import BashOperator dag = DAG( 'simple', schedule_interval='0 21 * * *', user_defined_macros={ 'next_execution_date': '{{ dag.following_schedule(execution_date) }}', }, ) task = BashOperator( task_id='bash_op', bash_command='echo "{{ next_execution_date }}"', dag=dag, ) The use case here is to back-port the new Airflow v1.8 next_execution_date macro to work in Airflow v1.7. Unfortunately, this template is rendered without macro

In my DAG file, I have define a on_failure_callback() function to post a Slack in case of failure. It works well if I specify for each operator in my DAG : on_failure_callback=on_failure_callback() Is there a way to automate (via default_args for instance, or via my DAG object) the dispatch to all of my operators? I finally found a way to do that. You can pass your on_failure_callback as a default_args class Foo: @staticmethod def get_default_args(): """ Return default args :return: default_args """ default_args = { 'on_failure_callback': Foo.on_failure_callback } return default_args

I have written a DAG with multiple PythonOperators task1 = af_op.PythonOperator(task_id='Data_Extraction_Environment', provide_context=True, python_callable=Task1, dag=dag1) def Task1(**kwargs): return(kwargs['dag_run'].conf.get('file')) From PythonOperator i am calling "Task1" method. That method is returning a value,that value i need to pass to the next PythonOperator.How can i get the value from the "task1" variable or How can i get the value which is returned from Task1 method? updated : def Task1(**kwargs): file_name = kwargs['dag_run'].conf.get[file] task_instance = kwargs['task_instance

The usual instructions for running Airflow do not apply on a Windows environment: # airflow needs a home, ~/airflow is the default, # but you can lay foundation somewhere else if you prefer # (optional) export AIRFLOW_HOME=~/airflow # install from pypi using pip pip install airflow # initialize the database airflow initdb # start the web server, default port is 8080 airflow webserver -p 8080 The Airflow utility is not available in the command line and I can't find it elsewhere to be manually added. How can Airflow run on Windows? You can activate bash in windows and follow the tutorial as is.

I am really a newbie in this forum. But I have been playing with airflow, for sometime, for our company. Sorry if this question sounds really dumb. I am writing a pipeline using bunch of BashOperators. Basically, for each Task, I want to simply call a REST api using 'curl' This is what my pipeline looks like(very simplified version): from airflow import DAG from airflow.operators import BashOperator, PythonOperator from dateutil import tz import datetime datetime_obj = datetime.datetime default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime.datetime.combine

官网: http://airflow.apache.org/installation.html 原理: https://www.cnblogs.com/cord/p/9450910.html 安装: https://www.cnblogs.com/cord/p/9226608.html 高可用部署等： https://www.jianshu.com/p/2ecef979c606 使用方法等: https://www.jianshu.com/p/cbff05e3f125 airflow: https://airflow.apache.org/ [root@node1 ~]# ps -ef |grep 10740 root 4417 10740 2 03:28 ? 00:00:02 [ready] gunicorn: worker [airflow-webserver] root 4719 10740 3 03:29 ? 00:00:02 [ready] gunicorn: worker [airflow-webserver] root 5069 10740 6 03:29 ? 00:00:02 [ready] gunicorn: worker [airflow-webserver] root 7426 10740 47 03:30 ? 00:00:02 [ready]