airflow

We are using Airflow v 1.9.0. We have 100+ dags and the instance is really slow. The scheduler is only launching some tasks. In order to reduce the amount of CPU usage, we want to tweak some configuration parameters, namely: min_file_process_interval and dag_dir_list_interval . The documentation is not really clear about the difference between the two min_file_process_interval : In cases where there are only a small number of DAG definition files, the loop could potentially process the DAG definition files many times a minute. To control the rate of DAG file processing, the min_file_process

I want to use Airflow package extras s3 and postgres in PyCharm but do not know how to install them (on macOS Sierra). My attempts so far Airflow itself can be installed from Preferences > Project > Project interpreter > + but not the extras as far as I can work out. The extras can be installed with pip in the terminal using $ pip install airflow[s3,postgres] but they end up in a different interpreter ( ~/anaconda ) than the one used by PyCharm ( /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7 ). Checking the Python executables in my /usr/local/bin directory I found

I noticed that for scheduled task the execution date is set in the past according to Airflow was developed as a solution for ETL needs. In the ETL world, you typically summarize data. So, if I want to summarize data for 2016-02-19, I would do it at 2016-02-20 midnight GMT, which would be right after all data for 2016-02-19 becomes available. however, when a dag triggers another dag the execution time is set to now(). Is there a way to have the triggered dags with the same execution time of triggering dag? Of course, I can rewrite the template and use yesterday_ds, however, this is a tricky

I am new to Airflow. I have come across a scenario, where Parent DAG need to pass some dynamic number (let's say n ) to Sub DAG. Where as SubDAG will use this number to dynamically create n parallel tasks. Airflow documentation doesn't cover a way to achieve this. So I have explore couple of ways : Option - 1(Using xcom Pull) I have tried to pass as a xcom value, but for some reason SubDAG is not resolving to the passed value. Parent Dag File def load_dag(**kwargs): number_of_runs = json.dumps(kwargs['dag_run'].conf['number_of_runs']) dag_data = json.dumps({ "number_of_runs": number_of_runs })

Good Morning. I'm trying to setup a DAG too Watch/sense for a file to hit a network folder Process the file Archive the file Using the tutorials online and stackoverflow I have been able to come up with the following DAG and Operator that successfully achieves the objectives, however I would like the DAG to be rescheduled or rerun on completion so it starts watching/sensing for another file. I attempted to set a variable max_active_runs:1 and then a schedule_interval: timedelta(seconds=5) this yes reschedules the DAG but starts queuing task and locks the file. Any ideas welcome on how I could