Run node.js database migrations on Google Cloud SQL during Google Cloud Build

Question

I would like to run database migrations written in node.js during the Cloud Build process.

Currently, the database migration command is being executed but it seems

Answer 1

In the case with Cloud SQL and Node.js it would look something like this:

steps:
  # Install Node.js dependencies
  - id: yarn-install
    name: node:dubnium
    entrypoint: yarn
    waitFor: ['-']

  # Install CLoud SQL proxy
  - id: proxy-install
    name: alpine:3
    entrypoint: sh
    args:
      - '-c'
      - 'wget -O /workspace/cloud_sql_proxy https://storage.googleapis.com/cloudsql-proxy/v1.16/cloud_sql_proxy.linux.386 && chmod +x /workspace/cloud_sql_proxy'
    waitFor: ['-']

  # Migrate database schema to the latest version
  # https://knexjs.org/#Migrations-CLI
  - id: migrate
    name: node:dubnium
    entrypoint: sh
    args:
      - '-c'
      - '(/workspace/cloud_sql_proxy -dir=/workspace -instances=<CLOUD_SQL_CONNECTION> & sleep 2) && yarn run knex migrate:latest'
    timeout: '1200s'
    waitFor: ['yarn-install', 'proxy-install']

timeout: '1200s'

You would launch yarn install and download Cloud SQL Proxy in parallel. Once these two steps are complete, you run launch the proxy, wait 2 seconds and finally run yarn run knex migrate:latest.

For this to work you would need Cloud SQL Admin API enabled in your GCP project.

Where <CLOUD_SQL_INSTANCE> is your Cloud SQL instance connection name that can be found here. The same name will be used in your SQL connection settings, e.g. host=/cloudsql/my-project/us-central1/db.

Also, make sure that the Cloud Build service account has "Cloud SQL Client" role in the GCP project, where the db instance is located.

Answer 2

As of tag 1.16 of gcr.io/cloudsql-docker/gce-proxy, the currently accepted answer no longer works. Here is a different approach that keeps the proxy in the same step as the commands that need it:

  - id: cmd-with-proxy
    name: [YOUR-CONTAINER-HERE]
    timeout: 100s
    entrypoint: sh
    args:
      - -c
      - '(/workspace/cloud_sql_proxy -dir=/workspace -instances=[INSTANCE_CONNECTION_NAME] & sleep 2) && [YOUR-COMMAND-HERE]'

The proxy will automatically exit once the main process exits. Additionally, it'll mark the step as "ERROR" if either the proxy or the command given fails.

This does require the binary is in the /workspace volume, but this can be provided either manually or via a prereq step like this:

  - id: proxy-install
    name: alpine:3.10
    entrypoint: sh
    args:
      - -c
      - 'wget -O /workspace/cloud_sql_proxy https://storage.googleapis.com/cloudsql-proxy/v1.16/cloud_sql_proxy.linux.386 &&  chmod +x /workspace/cloud_sql_proxy'

Additionally, this should work with TCP since the proxy will be in the same container as the command.

Answer 3

Use google-appengine/exec-wrapper. It is an image to do exactly this. Usage (see README in link):

steps:
- name: "gcr.io/google-appengine/exec-wrapper"
  args: ["-i", "gcr.io/my-project/appengine/some-long-name",
         "-e", "ENV_VARIABLE_1=value1", "-e", "ENV_2=value2",
         "-s", "my-project:us-central1:my_cloudsql_instance",
         "--", "bundle", "exec", "rake", "db:migrate"]

The -s sets the proxy target.

Answer 4

Here's how to combine Cloud Build + Cloud SQL Proxy + Docker.

If you're running your database migrations/operations within a Docker container in Cloud Build, it won't be able to directly access your proxy, because Docker containers are isolated from the host machine.

Here's what I managed to get up and running:

  - id: build
    # Build your application
    waitFor: ['-']

  - id: install-proxy
    name: gcr.io/cloud-builders/wget
    entrypoint: bash
    args:
      - -c
      - wget -O /workspace/cloud_sql_proxy https://storage.googleapis.com/cloudsql-proxy/v1.15/cloud_sql_proxy.linux.386 && chmod +x /workspace/cloud_sql_proxy
    waitFor: ['-']

  - id: migrate
    name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - |
        /workspace/cloud_sql_proxy -dir=/workspace -instances=projectid:region:instanceid & sleep 2 && \
        docker run -v /workspace:/root \
        --env DATABASE_HOST=/root/projectid:region:instanceid \
        # Pass other necessary env variables like db username/password, etc.
        $_IMAGE_URL:$COMMIT_SHA
    timeout: '1200s'
    waitFor: [build, install-proxy]

Because our db operations are taking place within the Docker container, I found the best way to provide the access to Cloud SQL by specifying the Unix socket -dir/workspace instead of exposing a TCP port 5432.

Note: I recommend using the directory /workspace instead of /cloudsql for Cloud Build.

Then we mounted the /workspace directory to Docker container's /root directory, which is the default directory where your application code resides. When I tried to mount it to other than /root, nothing seemed to happen (perhaps a permission issue with no error output).

Also: I noticed the proxy version 1.15 works well. I had issues with newer versions. Your mileage may vary.

Answer 5

Cloud Build runs using a service account and it looks like you need to grant access to Cloud SQL for this account. You can find additional info about setting service account permissions here.