问题
Using Lambda to move files from an S3 to our Redshift.
The data is placed in the S3 using an UNLOAD command directly from the data provider's Redshift. It comes in 10 different parts that, due to running in parallel, sometimes complete at different times.
I want the Lambda trigger to wait until all the data is completely uploaded before firing the trigger to import the data to my Redshift.
There is an event option in Lambda called "Complete Multipart Upload." Does the UNLOAD function count as a multipart upload within Lambda? Or would the simple "POST" event not fire until all the parts are completely uploaded by the provider?
There is no explicit documentation confirming that Redshift's UNLOAD command counts as a Multipart upload, or any confirming that the trigger will not fire until the data provider's entire upload is complete.
回答1:
For Amazon S3, a multi-part upload is a single file, uploaded to S3 in multiple parts. When all parts have been uploaded, the client calls CompleteMultipartUpload. Only after the client calls CompleteMultipartUpload will the file appear in S3.
And only after the file is complete will the Lambda function be triggered. You will not get a Lambda trigger for each part.
If your UNLOAD operation is generating multiple objects/files in S3, then it is NOT an S3 "multi-part upload".
来源：https://stackoverflow.com/questions/43987732/aws-lambda-and-multipart-upload-to-from-s3