Access AWS S3 from Lambda within VPC

Question

Overall, I\'m pretty confused by using AWS Lambda within a VPC. The problem is Lambda is timing out while trying to access an S3 bucket. The solution seems to be a VPC Endpo

Answer 1

With boto3, the S3 urls are virtual by default, which then require internet access to be resolved to region specific urls. This causes the hanging of the Lambda function until timeout.

To resolve this requires use of a Config object when creating the client, which tells boto3 to create path based S3 urls instead:

import boto3 
import botocore

client = boto3.client('s3', 'ap-southeast-2', config=botocore.config.Config(s3={'addressing_style':'path'}))

Note that the region in the call must be the region to which you are deploying the lambda and VPC Endpoint.

Then you will be able to use the pl-xxxxxx prefix list for the VPC Endpoint within the Lambda's security group, and still access S3.

Here is a working CloudFormation script that demonstrates this. It creates an S3 bucket, a lambda (that puts records into the bucket) associated to a VPC containing only private subnets and the VPC Endpoint, and necessary IAM roles.