apache-airflow

I'm trying to make an Airflow task intentionally fail and error out by passing in a Bash line ( thisshouldnotrun ) that doesn't work. Airflow is outputting the following: [2017-06-15 17:44:17,869] {bash_operator.py:94} INFO - /tmp/airflowtmpLFTMX7/run_bashm2MEsS: line 7: thisshouldnotrun: command not found [2017-06-15 17:44:17,869] {bash_operator.py:97} INFO - Command exited with return code 127 [2017-06-15 17:44:17,869] {models.py:1417} ERROR - Bash command failed Traceback (most recent call last): File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/models.py", line 1374, in run

Below is the config im using [core] # The home folder for airflow, default is ~/airflow airflow_home = /root/airflow # The folder where your airflow pipelines live, most likely a # subfolder in a code repository dags_folder = /root/airflow/dags # The folder where airflow should store its log files. This location base_log_folder = /root/airflow/logs # An S3 location can be provided for log backups # For S3, use the full URL to the base folder (starting with "s3://...") s3_log_folder = None # The executor class that airflow should use. Choices include # SequentialExecutor, LocalExecutor,

How can I configure Airflow so that any failure in the DAG will (immediately) result in a slack message? At this moment I manage it by creating a slack_failed_task: slack_failed_task = SlackAPIPostOperator( task_id='slack_failed', channel="#datalabs", trigger_rule='one_failed', token="...", text = ':red_circle: DAG Failed', icon_url = 'http://airbnb.io/img/projects/airflow3.png', dag=dag) And set this task (one_failed) upstream from each other task in the DAG: slack_failed_task << download_task_a slack_failed_task << download_task_b slack_failed_task << process_task_c slack_failed_task <<

I am not able see the logs attached to the tasks from the Airflow UI: Log related settings in airflow.cfg file are: remote_base_log_folder = base_log_folder = /home/my_projects/ksaprice_project/airflow/logs worker_log_server_port = 8793 child_process_log_directory = /home/my_projects/ksaprice_project/airflow/logs/scheduler Although I am setting remote_base_log_folter it is trying to fetch the log from http://:8793/log/tutorial/print_date/2017-08-02T00:00:00 - I don't understand this behavior. According to the settings the workers should store the logs at /home/my_projects/ksaprice_project

I am using a LocalExecutor and my dag has 3 tasks where task(C) is dependant on task(A). Task(B) and task(A) can run in parallel something like below A-->C B So task(A) has failed and but task(B) ran fine . Task(C) is yet to run as task(A) has failed. My question is how do i re run Task(A) alone so Task(C) runs once Task(A) completes and Airflow UI marks them as success. In the UI: Go to the dag, and dag run of the run you want to change Click on GraphView Click on task A Click "Clear" This will let task A run again, and if it succeeds, task C should run. This works because when you clear a