Unable to access BigQuery from local App Engine development server

Question

This is specifically a question relating to server to server authorisation between a python Google AppEngine app and Google\'s BigQuery, but could be relevant for other clou

Answer 1

So searching deeper for PyCrypto and local appengine sandbox lead me onto this thread and response specifically...

https://code.google.com/p/googleappengine/issues/detail?id=1627#c22

This is fixed in 1.7.4. However, you must use easy_install -Z (--always-unzip) to install PyCrypto. The default zipfile option in OSX 10.8 is incompatible with the sandbox emulation in the dev_appserver.

The solution turns out to be very straight forward...

I used:

sudo easy_install pycrypto

and it should have been:

sudo easy_install -Z pycrypto

as per the thread above. Using PIP will work as well:

pip install pycrypto 

or a manual download and install of pycrypto will also work. I tested all three.

If you have installed pycrypto with easy_install and without -Z flag then you may want to install pip just so you can easily uninstall pycrypto...

easy_install pip

for the record I built and installed libgmp, as pil and the manual install showed this warning...

warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.

Although this gave me fastmath, it was not essential to solve the problem as the Crypto libs gracefully fail to slowmath.

Another point that tripped me up for a bit was I removed pycrypto from app.yaml whilst testing to see if OpenSSL might give me all I need.

So dont forget to add...

- name: pycrypto
  version: latest

into app.yaml under the libraries: section.

With this missing the native _counter library was not imported hence Counter failed etc.

Also for the record any talk of having to move Crypto into the app folders themselves or out of the default Mac OS X location of /Library/Python/2.7/site-packages/Crypto was only valid in earlier versions of the dev server.

Similarly there is now no need to edit any _WHITE_LIST_C_MODULES lists (which is in sandbox.py in appengine 1.8 onwards, which also includes the regex which allows Crypto.Util._counter etc)

The other bit of the puzzle in case you get here before discovering the key issue is that the key file you download from the console is PKCS12 and is downloaded as hex text, so I converted that to binary and then converted that to a PEM so I could include it in the source code.