configuring Airflow to work with CeleryExecutor

问题

I try to configure Airbnb AirFlow to use the CeleryExecutor like this:

I changed the executer in the airflow.cfg from SequentialExecutor to CeleryExecutor:

# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = CeleryExecutor

But I get the following error:

airflow.configuration.AirflowConfigException: error: cannot use sqlite with the CeleryExecutor

Note that the sql_alchemy_conn is configured like this:

sql_alchemy_conn = sqlite:////root/airflow/airflow.db

I looked at Airflow's GIT (https://github.com/airbnb/airflow/blob/master/airflow/configuration.py)

and found that the following code throws this exception:

def _validate(self):
        if (
                self.get("core", "executor") != 'SequentialExecutor' and
                "sqlite" in self.get('core', 'sql_alchemy_conn')):
            raise AirflowConfigException("error: cannot use sqlite with the {}".
                format(self.get('core', 'executor')))

It seems from this validate method that the sql_alchemy_conn cannot contain sqlite.

Do you have any idea how to configure the CeleryExecutor without sqllite? please note that I downloaded rabitMQ for working with the CeleryExecuter as required.

回答1:

It is said by AirFlow that the CeleryExecutor requires other backend than default database SQLite. You have to use MySQL or PostgreSQL, for example.

The sql_alchemy_conn in airflow.cfg must be changed to follow the SqlAlchemy connection string structure (see SqlAlchemy document)

For example,

sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@127.0.0.1:5432/airflow

回答2:

To configure Airflow for mysql firstly install mysql this might help or just google it

goto airflow installation director usually /home//airflow
edit airflow.cfg
locate

sql_alchemy_conn = sqlite:////home/vipul/airflow/airflow.db

and add # in front of it so it looks like

#sql_alchemy_conn = sqlite:////home/vipul/airflow/airflow.db

if you have default sqlite

add this line below

sql_alchemy_conn = mysql://:@localhost:3306/
save the file
run command

airflow initdb

and done !

回答3:

As other answers have stated you need to use a different database besides SQLite. Additionally you need to install rabbitmq, configure it appropriately, and change each of your airflow.cfg's to have the correct rabbitmq information. For an excellent tutorial on this see A Guide On How To Build An Airflow Server/Cluster.

回答4:

If you run it on a kubernetes cluster. Use the following config:

airflow:
  config:
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://postgres:airflow@airflow-postgresql:5432/airflow

来源：https://stackoverflow.com/questions/36822515/configuring-airflow-to-work-with-celeryexecutor

标签

airflow