airflow

来源： https://stackoverflow.com/questions/62686753/airflow-dag-id-could-not-be-found-issue-when-using-kubernetes-executor

问题 Is there any way to get the exception details on the airflow on_failure_callback ? I've noticed it's not part of context . I'd like to create a generic exception handling mechanism which posts to Slack information about the errors, including details about the exception. I've now managed to trigger/execute the callback and post to Slack, but can't post the exception details. Thanks. 回答1: The error is added to the context here. So you can actually get that simply by doing: context.get(

问题 I am using apache airflow for running my dags. I am getting an exception as: *** Log file does not exist: /opt/airflow/logs/download2/download2/2020-07-26T15:00:00+00:00/1.log *** Fetching from: http://fb3393f5f01e:8793/log/download2/download2/2020-07-26T15:00:00+00:00/1.log *** Failed to fetch log file from worker. HTTPConnectionPool(host='fb3393f5f01e', port=8793): Max retries exceeded with url: /log/download2/download2/2020-07-26T15:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3

问题 I am using apache airflow for running my dags. I am getting an exception as: *** Log file does not exist: /opt/airflow/logs/download2/download2/2020-07-26T15:00:00+00:00/1.log *** Fetching from: http://fb3393f5f01e:8793/log/download2/download2/2020-07-26T15:00:00+00:00/1.log *** Failed to fetch log file from worker. HTTPConnectionPool(host='fb3393f5f01e', port=8793): Max retries exceeded with url: /log/download2/download2/2020-07-26T15:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3

问题 NOTE: similar question asked here, but no proper solution provided. I setted up an EKS cluster via eksctl tool with single EC2 node. Deploy a Pod inside the EC2 node, this Pod writes the logs into s3 bucket. All worked fine when I used IAM user with key and secret. But now I want this Pod to use IAM Role instead. This Pod uses a newly created role with AmazonS3FullAccess permissions named prod-airflow-logs . According to the Docs, I also added "ec2.amazonaws.com" in this role's trust

问题 I have static html documentation built using sphinx in: $AIRFLOW_HOME/plugins/docs/ I want to create a new menu link "My Documentation" in the Airflow UI so this works: class DocsView(BaseView): @expose("/") def my_docs(self): return send_from_directory(os.path.abspath("plugins/docs/build/html"), 'index.html') docs_view = DocsView( category="My Documentation", name="Plugins", endpoint="my_docs" ) And in my custom plugin class: class MyPlugin(AirflowPlugin): admin_views = [docs_view] The link

问题 I have static html documentation built using sphinx in: $AIRFLOW_HOME/plugins/docs/ I want to create a new menu link "My Documentation" in the Airflow UI so this works: class DocsView(BaseView): @expose("/") def my_docs(self): return send_from_directory(os.path.abspath("plugins/docs/build/html"), 'index.html') docs_view = DocsView( category="My Documentation", name="Plugins", endpoint="my_docs" ) And in my custom plugin class: class MyPlugin(AirflowPlugin): admin_views = [docs_view] The link

问题 I have static html documentation built using sphinx in: $AIRFLOW_HOME/plugins/docs/ I want to create a new menu link "My Documentation" in the Airflow UI so this works: class DocsView(BaseView): @expose("/") def my_docs(self): return send_from_directory(os.path.abspath("plugins/docs/build/html"), 'index.html') docs_view = DocsView( category="My Documentation", name="Plugins", endpoint="my_docs" ) And in my custom plugin class: class MyPlugin(AirflowPlugin): admin_views = [docs_view] The link

问题 Requirement: Edit the S3 file for the last row and remove double-quotes and extra pipeline and upload it back the same file back to s3 path Operator cleanup = S3FileTransformOperator( task_id='cleanup', source_s3_key='s3://path/outbound/incoming.txt', dest_s3_key='s3://path/outbound/incoming.txt', replace=True, transform_script='/usr/local/airflow/dags/scripts/clean_up.py' ) > Method 1 Approach Issue: Able to run the python script locally and but while running in Airflow it threw an error as

问题 How can I get an execution_date parameter in outside of dag? execution_min = "{{execution_date.strftime('%M') }}" if execution_min == '00': logging.info('**** ' + "YES, It's 00") final_task = DummyOperator( task_id='task_y00', ... dag=dag ) else: logging.info('**** ' + "NOPE!!!") final_task = DummyOperator( task_id='task_n00', ... dag=dag ) I want to set a task stream with dynamically with execution_date (especially minute) But Jinja template won't work with template_fields = ['execution_date