airflow

问题 I am having a tough time in figuring out how to find the failed task for the same dag run running twice on same day( same execution day ). Consider an example when a dag with dag_id=1 has failed on the first run (due to any reason lets say connection timeout maybe) and task got failed. TaskInstance table will contain the entry of the failed task when we try to query it. GREAT!! But, If I re-run the same dag(note that dag_id is still 1) then in the last task(it has the rule of ALL_DONE so

问题 I am having a tough time in figuring out how to find the failed task for the same dag run running twice on same day( same execution day ). Consider an example when a dag with dag_id=1 has failed on the first run (due to any reason lets say connection timeout maybe) and task got failed. TaskInstance table will contain the entry of the failed task when we try to query it. GREAT!! But, If I re-run the same dag(note that dag_id is still 1) then in the last task(it has the rule of ALL_DONE so

问题 I am having a tough time in figuring out how to find the failed task for the same dag run running twice on same day( same execution day ). Consider an example when a dag with dag_id=1 has failed on the first run (due to any reason lets say connection timeout maybe) and task got failed. TaskInstance table will contain the entry of the failed task when we try to query it. GREAT!! But, If I re-run the same dag(note that dag_id is still 1) then in the last task(it has the rule of ALL_DONE so

问题 The current issues I am having is that LDAP settings do not seem to work with Airflow. My current code does not produce a login screen nor are there logs in the docker container that shows it is attempting to connect to the LDAP server. I have attempted to follow these tutorials but they appear to do nothing (i.e. the webserver_config.py is not being used) TutorialOne TutorialTwo Any information regarding how the connection works would be greatly appreciated! webserver_config.py : import os

问题 I have setup Airflow a SQL Server as backend (SQL Azure). Init DB is successful. I am trying to run a simple dag every 2 minutes. The dag has 2 tasks: print date sleep When it start the airflow scheduler, it creates tasks instances for both the tasks, the first one succeeds & the second one seems to be stuck in "running" state. Looking at scheduler logs, I see the following error repeatedly. [2019-01-04 11:38:48,253] {jobs.py:397} ERROR - Got an exception! Propagating... Traceback (most

问题 I have setup Airflow a SQL Server as backend (SQL Azure). Init DB is successful. I am trying to run a simple dag every 2 minutes. The dag has 2 tasks: print date sleep When it start the airflow scheduler, it creates tasks instances for both the tasks, the first one succeeds & the second one seems to be stuck in "running" state. Looking at scheduler logs, I see the following error repeatedly. [2019-01-04 11:38:48,253] {jobs.py:397} ERROR - Got an exception! Propagating... Traceback (most

问题 I'm trying to access the Airflow Providers, specifically the AWS providers, found here I'm building a docker image and installing Airflow using PIP and including the AWS subpackage in the install command. pip install 'apache-airflow[crypto,aws,celery,postgres,hive,jdbc,mysql,ssh]==1.10.9' \ However, i'm unable to access the Provider from Python. from airflow.providers.amazon.aws.hooks.glue import AwsGlueJobHook >>> from airflow.providers.amazon.aws.hooks.glue import * Traceback (most recent

问题 I'm trying to access the Airflow Providers, specifically the AWS providers, found here I'm building a docker image and installing Airflow using PIP and including the AWS subpackage in the install command. pip install 'apache-airflow[crypto,aws,celery,postgres,hive,jdbc,mysql,ssh]==1.10.9' \ However, i'm unable to access the Provider from Python. from airflow.providers.amazon.aws.hooks.glue import AwsGlueJobHook >>> from airflow.providers.amazon.aws.hooks.glue import * Traceback (most recent

问题 I have implemented email alerts on success and failure using on_success_callback and on_failure_callback. According to Airflow documentation, a context dictionary is passed as a single parameter to this function. How can I pass another parameter to these callback methods? Here is my code from airflow.utils.email import send_email_smtp def task_success_alert(context): subject = "[Airflow] DAG {0} - Task {1}: Success".format( context['task_instance_key_str'].split('__')[0], context['task

问题 I have implemented email alerts on success and failure using on_success_callback and on_failure_callback. According to Airflow documentation, a context dictionary is passed as a single parameter to this function. How can I pass another parameter to these callback methods? Here is my code from airflow.utils.email import send_email_smtp def task_success_alert(context): subject = "[Airflow] DAG {0} - Task {1}: Success".format( context['task_instance_key_str'].split('__')[0], context['task