Store and access password using Apache airflow

Question

We are using airflow as a scheduler. I want to invoke a simple bash operator in a DAG. The bash script needs password as an argument to do further processing.

How c

Answer 1

In this case I would use a PythonOperator from which you are able to get a Hook on your database connection using hook = PostgresHook(postgres_conn_id=postgres_conn_id). You can then call get_connection on this hook which will give you a Connection object from which you can get the host, login and password for your database connection.

Finally, use for example subprocess.call(your_script.sh, connection_string) passing the connection details as a parameter.

This method is a bit convoluted but it does allow you to keep the encryption for database connections in Airflow. Also, you should be able to pull this strategy into a separate Operator class inheriting the base behaviour from PythonOperator but adding the logic for getting the hook and calling the bash script.