What is solution for downloading arbitrarily large files to Cloud Storage using Google App Engine php55 or php7?

Question

I have a google app engine php55 service that periodically checks a public website and downloads a file. This file is typically small (<1MB). My simple app is based on the fo

Answer 1

Use Google Storage Transfer Service (STS). It can be called from the Google Cloud SDK from your existing App Engine application, and will transfer the files directly from S3 to GCS without hitting any of yout App Engine limits. Based on your description, I believe it meets your requirements:

• No data transfer limit
• Minimal code changes
• "Serverless" and simple to configure

STS has some additional benefits:

• Zero runtime cost. That is, App Engine simply makes the STS API call to start the transfer job which is handled by STS, so you're not billed for the time GAE would normally use to download/upload the files itself.
• You can go even more serverless by invoking STS from a Cloud Function trigged by Cloud Scheduler. I doubt you'd save much on costs, but it sure would be a neat setup.

The GCP docs have a guide on How to set up a Transfer from Amazon S3 to Cloud Storage.

---

Additional notes:

• You can test out Storage Transfer Service directly in the GCP Console and see if it's right for you before you decide on whether you make any major code or architecture decisions.