Syncing Git repo to Google Cloud

问题

So suppose I have a git repository https://github.com/jc/ and I have a location for the google bucket gs://acme-sales/.

Is there a way to write a python program which updates the changes which have been made in github and sync them to google cloud each time we run it?

I suppose we have to use gitpython to read the file from github link but how do I just keep updating the files to google bucket.

回答1:

If you don't need the sync to be immediate (i.e., w/in seconds), you could set up a cron job to periodically pull down the .zip archive of your repo, and upload it to Google Cloud Storage.

In your app.yaml:

runtime: python37

In your cron.yaml:

cron:
- description: "sync from git repo"
  url: /tasks/sync
  schedule: every 1 minute

In your main.py:

from urllib.request import urlopen
from zipfile import ZipFile

from flask import Flask
from google.cloud import storage

app = Flask(__name__)

client = storage.Client(project='your-project-name')
bucket = client.get_bucket('your-bucket-name')

# Path to the archive of your repository's master branch
repo = 'https://github.com/your-username/your-repo-name/archive/master.zip'

@app.route('/tasks/sync')
def sync():
    with ZipFile(BytesIO(urlopen(repo).read())) as zipfile:
        for filename in zipfile.namelist():
            blob = storage.Blob(filename, bucket)
            blob.upload_from_string(zipfile.read(filename))

Deploy with:

$ gcloud app deploy
$ gcloud app deploy cron.yaml

回答2:

I recommend you to use Google Cloud Build. It lets you syncing your repo and using gsutil you can automatically update your storage i.e gs://acme-sales on each commits to your GitHub repository:

steps:
- name: gcr.io/cloud-builders/gsutil
  args: ['cp', '/workspace', 'gs://acme-sales']

You may start on GitHub by connecting your repository i.e. https://github.com/jc/<repo_name> to Google Cloud Build using the link below:

https://github.com/apps/google-cloud-build

回答3:

Adding to the previous answer[s].

You can an option to "Automatically Mirror from GitHub or Bitbucket", which is self descriptive. Or you can use an "Automatic Build Trigger" directly with custom build steps to perform certain actions.

来源：https://stackoverflow.com/questions/52191307/syncing-git-repo-to-google-cloud

标签

python

google-cloud-platform

google-cloud-storage

GitPython