airflow

问题 Is it possible to make an Airflow DAG fail if any task fails? I usually have some cleaning up tasks at the end of a DAG and as it is now, whenever the last task succeeds the whole DAG is marked as a success. 回答1: Another solution can be to add a final PythonOperator that checks the status of all tasks in this run: final_status = PythonOperator( task_id='final_status', provide_context=True, python_callable=final_status, trigger_rule=TriggerRule.ALL_DONE, # Ensures this task runs even if

问题 I have configured Airflow with rabbitmq broker, the services: airflow worker airflow scheduler airflow webserver are running without any errors. The scheduler is pushing the tasks to execute on default rabbitmq queue: Even I tried airflow worker -q=default - worker still not receiving tasks to run. My airflow.cfg settings file: [core] # The home folder for airflow, default is ~/airflow airflow_home = /home/my_projects/ksaprice_project/airflow # The folder where your airflow pipelines live,

问题 With the help of this Stackoverflow post I just made a program (the one shown in the post) where when a file is placed inside an S3 bucket a task in one of my running DAGs is triggered and then I perform some work using the BashOperator. Once it's done though the DAG is no longer in a running state but instead goes into a success state and if I want to have it pick up another file I need to clear all the 'Past', 'Future', 'Upstream', 'Downstream' activity. I would like to make this program so

问题 I am getting the following error : File "<unknown>", line 1, in template TemplateSyntaxError: unexpected char u'\\' at 205 When I include this in my code: '{{ macros.ds_format(macros.ds_add(ds, -13), "%Y-%m-%d", "%Y%m%d") }}' I re-typed it in vim in case it was an error with encoding, but still no luck! 来源： https://stackoverflow.com/questions/44190948/templatesyntaxerror-unexpected-char-u-at-205-airflow-macros

问题 I have DAG where max_active_runs is set to 2, but now I want to run backfills for 20ish runs. I actually expected airflow to sort of schedule all the backfills but only start 2 at a time, but that doesn't seem to happen. When I run the backfill command it starts two, but the command doesn't return since it didn't manage to start them all, instead, it keeps on trying until it succeeds. So what I expected was this: I ran the backfill command All the runs are marked as running Command returns

问题 created an image contain /airflow/xcom/return.json with chmod +x on all sub-dir Since the log show it cannot find file or directory(tried chmod +x) strtpodbefore = KubernetesPodOperator(namespace='rguonew', image="mydockerimage", name="fail", image_pull_policy="Always", task_id="failing-task", get_logs= True, xcom_push=True, dag=dag )' Here is the log [2019-03-18 20:32:07,007] {logging_mixin.py:95} INFO - [2019-03-18 20:32:07,007] {pod_launcher.py:166} INFO - Running command... cat /airflow

问题 I have 3 tasks, A, B and C. I want to run task A only once, and then run task B monthly until end_date, then run task C only once to clean up. This is similar to this question, but not applicable. How to handle different task intervals on a single Dag in airflow? Thanks for your help 回答1: For task A that is supposed to run only once, you can take inspiration from here As far as tasks B & C are concerned, they can be tied up with A using a ShortCircuitOperator (as already told in the link you

问题 I have created a on_failure_callback function(refering Airflow default on_failure_callback) to handle task's failure. It works well when there is only one task in a DAG, however, if there are 2 more tasks, a task is randomly failed since the operator is null, it can resume later by manully . In airflow-scheduler.out the log is: [2018-05-08 14:24:21,237] {models.py:1595} ERROR - Executor reports task instance %s finished (%s) although the task says its %s. Was the task killed externally?

问题 I am trying to send parameter to an airflow task in order to identify the last execution. The following code always send {"try_number": "1"} as POST data. Airflow version: 1.10.2 Thanks xxx = SimpleHttpOperator( task_id='XXX', endpoint='/backend/XXX', http_conn_id='backend_url', data=json.dumps({"try_number": "{{ti.try_number}}"}), headers={"Content-Type": "application/json"}, response_check=lambda response: response.json().get('status') == 'ok', dag=dag, ) 回答1: The problem is with the

问题 I am scheduling dag in airflow for 10 minutes is not doing anything. here is my dags code: from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime, timedelta default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date':datetime.now(), 'email': ['airflow@airflow.com'], 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), } dag = DAG('Python_call', default_args=default_args,