airflow

I have an issue where the BashOperator is not logging all of the output from wget. It'll log only the first 1-5 lines of the output. I have tried this with only wget as the bash command: tester = BashOperator( task_id = 'testing', bash_command = "wget -N -r -nd --directory-prefix='/tmp/' http://apache.cs.utah.edu/httpcomponents/httpclient/source/httpcomponents-client-4.5.3-src.zip", dag = dag) I've also tried this as part of a longer bash script that has other commands that follow wget. Airflow does wait for the script to complete before firing downstream tasks. Here's an example bash script:

I'm attempting to generate a set of dynamic tasks from a XCOM variable. In the XCOM I'm storing a list and I want to use each element of the list to dynamically create a downstream task. My use case is that I have an upstream operator that checks a sftp server for files and returns a list of file names matching specific criteria. I want to create dynamic downstream tasks for each of the file names returned. I've simplified it to the below, and while it works I feel like its not an idiomatic airflow solution. In my use case, I would write a python function that's called from a python operator

New to airflow. Trying to run the sql and store the result in a BigQuery table. Getting following error. Not sure where to setup the default_rpoject_id. Please help me. Error: Traceback (most recent call last): File "/usr/local/bin/airflow", line 28, in <module> args.func(args) File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 585, in test ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True) File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 53, in wrapper result = func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages

In airflow, everything is supposed to be UTC (which is not affected by DST). However, we have workflows that deliver things based on time zones that are affected by DST. An example scenario: We have a job scheduled with a start date at 8:00 AM Eastern and a schedule interval of 24 hours. Everyday at 8 AM Eastern the scheduler sees that it has been 24 hours since the last run, and runs the job. DST Happens and we lose an hour. Today at 8 AM Eastern the scheduler sees that it has only been 23 hours because the time on the machine is UTC, and doesn't run the job until 9AM Eastern, which is a late

参考: https://github.com/teamclairvoyant/airflow-scheduler-failover-controller 1.stop failover 2.stop scheduler 3.clear failover meta 4.start failover #启动master上的scheduler . /data/venv/bin/activate supervisorctl status supervisorctl stop airflow_failover supervisorctl stop airflow_scheduler scheduler_failover_controller clear_metadata supervisorctl start airflow_failover #failover命令 scheduler_failover_controller metadata #Get the Metadata from Metastore scheduler_failover_controller clear_metadata #Clear the Metadata in Metastore scheduler_failover_controller is_scheduler_running #Checks if the