What's the best way to back up data on Google App Engine?

Question

Google itself provides two solutions.

http://code.google.com/appengine/docs/python/tools/uploadingdata.html

One of these is new and experimental and requires you to run a separate command (and separately enter your username and password) for each kind of data you want to back up.

The other requires you to twice write out information on the structure of all the kinds of your data, information that is already implicit in your models file. (We've got 25 different kinds of data so I'm sensitive to this kind of stuff. Plus it will mean future changes will have to be made in 3 places.)

Then there's Aral Balkan's solution (google for "gaebar"), but his code on Github hasn't been updated in about a year, and he's additionally telling people to do some modification to App Engine internals (which seems risky, since they change in every version).

I think I'm leaning towards Google's non-experimental solution, but they all seem pretty bad.

Answer 1

Here is a more explicit description of bulkloader.py no-config backup and restore solution. I do not know when this issue (stack overflow) arises.

Dumping data from fooapp into a file called dump1.bin:

python2.5 /usr/local/google_appengine/bulkloader.py  \
  --dump --url http://fooapp.appspot.com/remote-api-url \
  --filename dump1.bin

You must have the remote API enabled. The remote-api-url might be _ah/remote_api depending on your configuration.

Loading up data from dump1.bin to a local dev instance:

python2.5 /usr/local/google_appengine/bulkloader.py \
  --restore --url http://localhost:8080/remote-api-url \
  --filename dump1.bin --application fooapp

This assumes a UNIX-like system and that appengine is installed in /usr/local/google_appengine.

I don't know if it works with Java, but it might.

Answer 2

Have a look at AppRocket (an open-source replication engine that synchronizes Google App Engine datastore and MySQL database.) The project seems to be active.

Answer 3

What's wrong with the --dump functionality? Yes, you have to download each kind separately, but that's going to be the case with any solution. If you just want backups, it fits your requirements exactly.