How High Is the s3 File Size Download Limit If I Move My Google App Engine Service to an Amazon Lambda Function?

本小妞迷上赌 提交于 2019-12-11 12:15:09

问题


Broadly, I seek to answer the question, "Should I redevelop my app with the Amazon Web Service (AWS) platform?"

As described in another post, I have a google app engine (GAE) php55 service (which I am now contemplating migrating to AWS) that periodically checks a public web server and downloads a file directly to a Google Cloud Storage bucket. This file is typically small (<1MB), but occasionally over the 32MB individual response size limit on GAE. My simple app is based on the following:

<?php
$strSource = 'https://example.com/file.zip';

$strBucket = 'bucket-1234';
$strDirectory = '/path/to/file/'; // Google Cloud Storage directory
$strName = 'file.zip';
$strDestination = 'gs://' . $strBucket . '.appspot.com' . $strDirectory . $strName;

copy($strSource,$strDestination);
?>

My service needs to run about 100 times a day; often the files my service downloads are <<1MB and only about 1% of the time is the file over Google's current 32MB response size limit. I need the bucket to hold <1TB of data.

I am hesitant to implement Google's recommended solution (which is to run a Google Compute Engine virtual machine) because I would like to keep everything serverless.

If I redevelop this service as an AWS Lambda function (instead of GAE), will that immediately increase my app's individual file size limit up to 512MB?

(Note: The 512MB file size limit value assumes my Lambda function first downloads the file to /tmp, so long as I am diligent enough to clear the contents of this directory after transferring each file to an s3 bucket.)

Or is the Lambda function response size limit even higher and only based on what the Lambda function can download in the 15-minute execution time limit if I download directly to an s3 bucket?

My preliminary search suggests I should consider rebuilding in the AWS ecosystem, but I would like to avoid investing significant time learning a different ecosystem if a serverless solution for my needs on AWS is infeasible.

It also appears as though I would probably need to rewrite much of my current app in python so it is natively supported in AWS. Am I understanding these issues correctly? Are there any other quotas or limits that would restrict the size of a file I could transfer from a public web server into my s3 bucket using a Lambda function? Can I get around that 512MB limit by saving the file directly to the s3 bucket?

来源:https://stackoverflow.com/questions/58598052/how-high-is-the-s3-file-size-download-limit-if-i-move-my-google-app-engine-servi

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!