Run appengine backup from task queue

问题

In the 1.8.4 release of Google App Engine it states:

A Datastore Admin fix in this release improves security by ensuring that scheduled backups can now only be started by a cron or task queue task. Administrators can still start a backup by going to the Datastore Admin in the Admin Console.

The way to run scheduled backups with cron is documented, but how can we initiate backups from a task queue task?

Are there any other ways to programmatically run backup tasks?

回答1:

You can create a task queue task with method GET and URL "/_ah/datastore_admin/backup.create" with your parameters specified as arguments to the URL and target the task to run on the 'ah-builtin-python-bundle' version. Example:

url = '/_ah/datastore_admin/backup.create?filesystem=blobstore&name=MyBackup&kind=Kind1&kind=Kind2'
taskqueue.add(
        url=url,
        target='ah-builtin-python-bundle',
        method='GET',
        )

I have cron jobs that trigger my own handlers that then look up a config and create a task queue backup based on that config. This lets me change backup settings without having to update cron jobs and lets me have a lot more control.

The options you can specify in the URL are the same as the documentation describes for CRON job backups so you can specify namespace and gs-bucket-name as well.

I believe to do this in Java you have to create a queue with a target in the queue definition and add your tasks to that queue.

回答2:

I did this by combining Bryce' solution with the code from googles scheduled backup documentation. This way, I'm still using cron.yaml but I have the flexibility for differences in each environment (e.g. don't run a backup in the dev/stage branch based on config in the datastore, don't specify types in the URL that haven't made it out of dev yet).

I also was able to generate the &kind=xxx pairs using this:

    from google.appengine.ext.ndb import metadata
    backup_types = "".join(["&kind=" + kind for kind in metadata.get_kinds()  if not kind.startswith("_")])

The steps were pretty simple in retrospect.

Setup:

a) Enable your default cloud storage bucket

b) Enable datastore admin

Steps:

1. Add a cron job to kick off the backup (cron.yaml):

   cron:
   - description: daily backup
     url: /taskqueue-ds-backup/
     schedule: every 24 hours from 21:00 to 21:59
   

2. Add a queue to process the tasks (queue.yaml):

   - name: ds-backup-queue
     rate: 1/s
     retry_parameters:
       task_retry_limit: 1
   

3. Add a route to the task queue handler:

   routes = [...,
             RedirectRoute('/taskqueue-ds-backup/',
                tasks.BackupDataTaskHandler, 
                name='ds-backup-queue', strict_slash=True), ...]
   

4. Add the handler to process the enqueued items:

   from google.appengine.api import app_identity
   from google.appengine.api import taskqueue
   from google.appengine.ext.ndb import metadata
   import config
   
   class BackupDataTaskHandler(webapp2.RequestHandler):
       def get(self):
           enable_ds_backup = bool(config.get_config_setting("enable_datastore_backup", default_value="False"))
           if not enable_ds_backup:
               logging.debug("skipping backup. backup is not enabled in config")
               return
   
           backup_types = "".join(["&kind=" + kind for kind in metadata.get_kinds() if not kind.startswith("_")])
   
           file_name_prefix = app_identity.get_application_id().replace(" ", "_") + "_"
           bucket_name = app_identity.get_default_gcs_bucket_name()
   
           backup_url = "/_ah/datastore_admin/backup.create?name={0}&filesystem=gs&gs_bucket_name={1}{2}".format(file_name_prefix, bucket_name, backup_types)
           logging.debug("backup_url: " + backup_url)
   
           # run the backup as a service in production. note this is only possible if you're signed up for the managed backups beta.
           if app_identity.get_application_id() == "production-app-id":
               backup_url += "&run_as_a_service=T"
   
           taskqueue.add(
                   url=backup_url,
                   target='ah-builtin-python-bundle',
                   method='GET',
                   )
           logging.debug("BackupDataTaskHandler completed.")
   

来源：https://stackoverflow.com/questions/18792027/run-appengine-backup-from-task-queue