airflow

Wiring top-level DAGs together

由 可紊 提交于 2019-11-26 09:57:33
问题 I need to have several identical (differing only in arguments) top-level DAG s that can also be triggered together with following constraints / assumptions: Individual top-level DAGs will have schedule_interval=None as they will only need occasional manual triggering The series of DAGs, however, needs to run daily Order and number of DAGs in series is fixed (known ahead of writing code) and changes rarely (once in a few months) Irrespective of whether a DAG fails or succeeds, the chain of

How to submit Spark jobs to EMR cluster from Airflow?

由 女生的网名这么多〃 提交于 2019-11-26 04:51:17
问题 How can I establish a connection between EMR master cluster(created by Terraform) and Airflow. I have Airflow setup under AWS EC2 server with same SG,VPC and Subnet. I need solutions so that Airflow can talk to EMR and execute Spark submit. https://aws.amazon.com/blogs/big-data/build-a-concurrent-data-orchestration-pipeline-using-amazon-emr-and-apache-livy/ These blogs have understanding on execution after connection has been established.(Didn\'t help much) In airflow I have made a connection

H3C S5560交换机风扇反转告警

由 我的未来我决定 提交于 2019-11-26 04:26:58
时间： 2016-09-25 工程： 新疆农七师锦龙电力调度自动化升级项目 现场人员： 张健 问题描述： 现场所有的S5560系列交换机风扇信号灯黄灯闪烁，前面板SYS信号灯红色常亮。 现场环境： 检查交换机风扇配置信息发现交换机风扇的实际气流方向为：Airflow Direction: Port-to-power；而配置用户下看到风扇模块的优选气流方向为：Prefer Airflow Direction: Power-to-port，两者不一致（交换机风扇反转），导致交换机风扇状态为：FanDirectionFault，风扇信号灯告警，交换机SYS信号灯告警。 处理方法： 因为风扇模块的实际气流方向是无法改变的，只能去改变风扇模块的优选气流方向和风扇实际气流方向一致，操作步骤如下： [QIANZHI-B] [QIANZHI-B]dis fan Slot 1: Fan 1: State : FanDirectionFault //工作状态报错 Airflow Direction: Port-to-power //实际气流方向 Prefer Airflow Direction: Power-to-port //优选气流方向 Fan 2: State : FanDirectionFault Airflow Direction: Port-to-power Prefer Airflow

Airflow 1.9.0 is queuing but not launching tasks

由 不羁岁月 提交于 2019-11-26 04:24:55
问题 Airflow is randomly not running queued tasks some tasks dont even get queued status. I keep seeing below in the scheduler logs [2018-02-28 02:24:58,780] {jobs.py:1077} INFO - No tasks to consider for execution. I do see tasks in database that either have no status or queued status but they never get started. The airflow setup is running https://github.com/puckel/docker-airflow on ECS with Redis. There are 4 scheduler threads and 4 Celery worker tasks. For the tasks that are not running are
订阅 airflow