google-cloud-composer | 易学教程

Broken DAG: (…) No module named docker

阅读更多关于 Broken DAG: (…) No module named docker

问题 I have BigQuery connectors all running, but I have some existing scripts in Docker containers I wish to schedule on Cloud Composer instead of App Engine Flexible. I have the below script that seems to follow the examples I can find: import datetime from airflow import DAG from airflow import models from airflow.operators.docker_operator import DockerOperator yesterday = datetime.datetime.combine( datetime.datetime.today() - datetime.timedelta(1), datetime.datetime.min.time()) default_args = {

DAGs not clickable on Google Cloud Composer webserver, but working fine on a local Airflow

阅读更多关于 DAGs not clickable on Google Cloud Composer webserver, but working fine on a local Airflow

I'm using Google Cloud Composer (managed Airflow on Google Cloud Platform) with image version composer-0.5.3-airflow-1.9.0 and Python 2.7, and I'm facing a weird issue : after importing my DAGs, they are not clickable from the Web UI (and there are no buttons "Trigger DAG", "Graph view", ...), while all works perfectly when running a local Airflow. Even if non usable from the webserver on Composer, my DAGs still exist. I can list them using CLI ( list_dags ), describe them ( list_tasks ) and even trigger them ( trigger_dag ). Minimal example reproducing the issue A minimal example I used to

Airflow error importing DAG using plugin - Relationships can only be set between Operators

阅读更多关于 Airflow error importing DAG using plugin - Relationships can only be set between Operators

问题 I have written an airflow plugin that simply contains one custom operator (to support CMEK in BigQuery). I can create a simple DAG with a single task that uses this operator and that executes fine. However if I try and create a dependency in the DAG from a DummyOperator task to my custom operator task the DAG fails to load in the UI and throws the following error and I can't understand why this error is being thrown? Broken DAG: [/home/airflow/gcs/dags/js_bq_custom_plugin_v2.py] Relationships