Can't import Airflow plugins

Question

Following Airflow tutorial here.

Problem: The webserver returns the following error

Broken DAG: [/usr/local/airflow/dags/test_operat         


        

Answer 1

I encountered the same error while following these tutorials.

My fault, however, was that I had used space character ' ' in task_id, which isn't supported by Airflow.

Clearly the error didn't point towards the actual problem. Restarting both Airflow scheduler and webserver then showed the correct error message on WebUI.

Answer 2

In my case I managed to make a custom operator with the following steps:

1. Airflow 10.3
2. in DAG File from airflow.operators import MacrosPostgresOperator
3. In ~/airflow/plugins folder I have a python file custom_operator.py and the code is pretty simple

from airflow.plugins_manager import AirflowPlugin
from airflow.operators.postgres_operator import PostgresOperator

 class MacrosPostgresOperator(PostgresOperator):
    template_fields = ('sql', 'parameters')

class MacrosFirstPlugin(AirflowPlugin):
    name = "macros_first_plugin"
    operators = [MacrosPostgresOperator]

Answer 3

You must stop (CTRL-C) and restart your Airflow web server and scheduler.

Answer 4

I use airflow 1.10. If it's a custom operator that you want to import, you can upload it to the airflow plugins folder, and then in the DAG specify the import as :

from [filename] import [classname]

where : filename is the name of your plugin file classname is the name of your class.

For example : If the name of your file is my_first_plugin and name of the class is MyFirstOperator then, the import would be :

from my_first_plugin import MyFirstOperator

Worked for me as I am using airflow 1.10

Thanks ! Hope this helps !!

Answer 5

I restarted the webserver, and now everything works fine.

Here is what I think might have happened:

1. Before I started with the tutorial example, I tried running my own plugin and dag. There was a minor syntax error on the first run that I fixed, however after the fix I started getting the 'cannot import name' error.
2. I deleted the plugin and dag, and tried using the one from the tutorial to see what was going on.

My guess is that the error from step 1 somehow affected step 2.