airflow

Problem Is there any way in Airflow to create a workflow such that the number of tasks B.* is unknown until completion of Task A? I have looked at subdags but it looks like it can only work with a static set of tasks that have to be determined at Dag creation. Would dag triggers work? And if so could you please provide an example. I have an issue where it is impossible to know the number of task B's that will be needed to calculate Task C until Task A has been completed. Each Task B.* will take several hours to compute and cannot be combined. |---> Task B.1 --| |---> Task B.2 --| Task A ------

本人在centos7 的环境下部署，怎么在centos7 下配置静态 IP 关闭防火墙 以及安装jdk在这里不多赘述， centos7 配置静态ip可以参考: https://www.cnblogs.com/braveym/p/8523100.html 和 https://www.cnblogs.com/braveym/p/9096402.html Airflow 基础安装 1、默认自带python2环境，自行安装pip sudo yum -y install epel-release sduo yum install python-pip 2、进行pip的更新，否则很多安装会报错 sudo pip install --upgrade pip sudo pip install --upgrade setuptools 3、安装开发库 sudo yum install python-devel sudo yum install libevent-devel sudo yum install mysql-devel 4、安装centos7下的mysql 在安装mysql前建议先更新一下源，不然下载老失败 打开centos的yum文件夹（需要root权限或者拥有该目录的操作权限才可以） 输入命令cd /etc/yum.repos.d 用wget下载repo文件 输入命令sudo wget

Say you have an airflow DAG that doesn't make sense to backfill, meaning that, after it's run once, running it subsequent times quickly would be completely pointless. For example, if you're loading data from some source that is only updated hourly into your database, backfilling, which occurs in rapid succession, would just be importing the same data again and again. This is especially annoying when you instantiate a new hourly task, and it runs N amount of times for each hour it missed, doing redundant work, before it starts running on the interval you specified. The only solution I can think

Consider the following example of a DAG where the first task, get_id_creds , extracts a list of credentials from a database. This operation tells me what users in my database I am able to run further data preprocessing on and it writes those ids to the file /tmp/ids.txt . I then scan those ids into my DAG and use them to generate a list of upload_transaction tasks that can be run in parallel. My question is: Is there a more idiomatically correct, dynamic way to do this using airflow? What I have here feels clumsy and brittle. How can I directly pass a list of valid IDs from one process to that

when I put a new DAG python script in the dags folder, I can view a new entry of DAG in the DAG UI but it was not enabled automatically. On top of that, it seems does not loaded properly as well. I can only click on the Refresh button few times on the right side of the list and toggle the on/off button on the left side of the list to be able to schedule the DAG. These are manual process as I need to trigger something even though the DAG Script was put inside the dag folder. Anyone can help me on this ? Did I missed something ? Or this is a correct behavior in airflow ? By the way, as mentioned

I would like to create a conditional task in Airflow as described in the schema below. The expected scenario is the following: Task 1 executes If Task 1 succeed, then execute Task 2a Else If Task 1 fails, then execute Task 2b Finally execute Task 3 All tasks above are SSHExecuteOperator. I'm guessing I should be using the ShortCircuitOperator and / or XCom to manage the condition but I am not clear on how to implement that. Could you please describe the solution? You have to use airflow trigger rules All operators have a trigger_rule argument which defines the rule by which the generated task