How do I run custom python script in Google App engine

Question

Apologies for the long detailed question. Here goes...

The file has the name send_daily_report.py and uses some libraries which are detailed in a requirements.txt file.

My app.yaml looks as follows:

runtime: python27
threadsafe: false

handlers:
- url: /send_daily_report
  script: send_daily_report.py

libraries:
- name: ssl
  version: latest

My cron.yaml looks as follows:

cron:
- description: "Send unsent subscriptions to Glocell rewards every 1 minute"
  url: /send_daily_report
  schedule: every 1 day from 00:00

And finally my requirements.txt has the following contents:

Babel==2.5.3
cachetools==3.1.0
certifi==2017.11.5
chardet==3.0.4
coloredlogs==8.0
colorlog==3.1.0
et-xmlfile==1.0.1
google-api-core==1.10.0
google-auth==1.6.3
google-cloud-core==0.29.1
google-cloud-storage==1.15.0
google-resumable-media==0.3.2
googleapis-common-protos==1.5.10
humanfriendly==4.6
idna==2.6
jdcal==1.3
money==1.3.0
mysql-connector-python==8.0.5
mysqlclient==1.3.12
numpy==1.16.3
openpyxl==2.5.0
pandas==0.24.2
protobuf==3.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
python-dateutil==2.8.0
python-magic==0.4.15
pytz==2017.3
requests==2.18.4
rsa==4.0
six==1.12.0
urllib3==1.22

When I do a:

gcloud app deploy app.yaml cron.yaml

it seems to ignore my requirements.txt and doesn't install the dependent. I even ran it in the gcloud dev server (dev_appserver.py ./app.yaml --enable_console) locally and the libraries are definitely not being installed when I use the dev server console to try and import them.

I then tried to use Python 3.7. which actually installs the libraries in requirements.txt but the problem is in the app.yaml in the handlers I cannot specify the script name. For python 3.7 that value is always set to auto as per the gcloud app.yaml reference.

What do I need to do to execute my script in gcloud app engine cron. On a linux system this would have been a simple cronjob setup in the crontab as follows:

0 0 * * * python /send_daily_py

From what i'm reading about Python 3.7, it seems like I need to have django or flex to handle the requests and route them to execute my custom .py script which seems like overkill just so that I can execute one script. Surely there is a way around this?. I even tried to include the libraries in a subfolder within my app folder as follows:

pip install --upgrade -r requirements.txt -t ./lib

Then I added init.py within lib, and then I changed all my imports to use

import lib.name_of_library

Which did not work as those libraries could subsequently not import sub packages they need.

**

Heeeeelp!

**

Answer 1

App Engine is PaaS product, not an IaaS one (on which you could, indeed, just run a linux image and install the cron you mentioned). You cannot run arbitrary standalone python scrips in GAE. You might be able to achieve what you want by re-working the script to meet the GAE apps requirements - basically make the functionality callable from inside a HTTP(S) handler.

For the 1st generation standard environment (python27 runtime):

• the requirements.txt file isn't used by GAE. As you discovered, you can use it to vendor in your python dependencies, but there's more to do, see Copying a third-party library.
• your script functionality needs to be re-worked as a WSGI app, which is what you configure in your app.yaml. From Handlers element:

A script: directive must be a python import path, for example, package.module.app that points to a WSGI application. The last component of a script: directive using a Python module path is the name of a global variable in the module: that variable must be a WSGI app, and is usually called ** ** by convention.

Note: just like for a Python import statement, each subdirectory that is a package must contain a file named init.py.

• I'd suggest you go through Python Runtime Environment, your long list of requirements might not meet the sandbox limitations (in particular the Pure Python one).

For the 2nd generation standard environment (python37 runtime):

• dependencies are automatically installed from your requirements.txt file, see Specifying Dependencies
• only auto can be specified in a script: statement in app.yaml, because the app itself is specified via the entrypoint: config. So you need need to rework your script to be invoked as handler in that app. From Runtime and app elements:

For your app to receive HTTP requests, entrypoint should contain a command which starts a web server that listens on the port specified by the PORT environment variable.

The flexible environment (with similar re-work as for the 2nd gen standard one) could be a better fit, especially because you can configure instances with more ram/cpu resources (which you might need judging by your requirements.txt file) than in the standard environment.