Google Cloud Composer and Google Cloud SQL

Question

What ways do we have available to connect to a Google Cloud SQL (MySQL) instance from the newly introduced Google Cloud Composer? The intention is to get data from a Cloud S

Answer 1

"The Cloud SQL Proxy provides secure access to your Cloud SQL Second Generation instances without having to whitelist IP addresses or configure SSL." -Google CloudSQL-Proxy Docs

CloudSQL Proxy seems to be the recommended way to connect to CloudSQL above all others. So in Composer, as of release 1.6.1, we can create a new Kubernetes Pod to run the gcr.io/cloudsql-docker/gce-proxy:latest image, expose it through a service, then create a Connection in Composer to use in the operator.

To get set up:

• Follow Google's documentation

• Test the connection using info from Arik's Medium Post

  • Check that the pod was created kubectl get pods --all-namespaces

  • Check that the service was created kubectl get services --all-namespaces

  • Jump into a worker node kubectl --namespace=composer-1-6-1-airflow-1-10-1- exec -it airflow-worker- bash

    • Test mysql connection mysql -u composer -p --host .default.svc.cluster.local

Notes:

• Composer now uses namespaces to organize pods

• Pods in different namespaces don't talk to each other unless you give them the full path ..svc.cluster.local

• Creating a new Composer Connection with the full path will enable successful connection