How to import external scripts in a Airflow DAG with Python?

I have the following structure:

And I try to import the script inside some files of the inbound_layer like so:

import calc

However I get the following error message on Airflow web:

Any idea?

I needed insert the following script inside at the top of ren.py :

import sys, os
from airflow.models import Variable

DAGBAGS_DIR = Variable.get('DAGBAGS_DIR')
sys.path.append(DAGBAGS_DIR + '/bi/inbound_layer/')

This way I make available the current folder packages.

For airflow DAG, when you import your own module, you need make sure 2 things:

where is the module? You need to find where is the root path in you airflow folder. For example, in my dev box, the folders are:

~/projects/data/airflow/teams/team_name/projects/default/dags/dag_names/dag_files.py

The root is airflow, so if I put my modules my_module in

~/projects/data/airflow/teams/team_name/common

Then I need to use

from teams.team_name.common import my_module

In your case, if the root is the upper folder of bi, and you put the scripts of calc in bi/inbound_layer/test.py then you can use:

from bi.inbound_layer.test import calc

And you must make sure you have \__init\__.py files in the directory structure for the imports to function properly. You should have an empty file \__init\__.py in each folder in the path. It indicates this directory is part of airflow packages. In your case, you can use touch \__init\__.py (cli) under bi and _inbound_layer_ folders to create the empty __init\__.py.

来源：https://stackoverflow.com/questions/56134683/how-to-import-external-scripts-in-a-airflow-dag-with-python

标签