Upload image available at public URL to S3 using boto

Question

I\'m working in a Python web environment and I can simply upload a file from the filesystem to S3 using boto\'s key.set_contents_from_filename(path/to/file). However, I\'d l

Answer 1

Using the boto3 upload_fileobj method, you can stream a file to an S3 bucket, without saving to disk. Here is my function:

import boto3
import StringIO
import contextlib
import requests

def upload(url):
    # Get the service client
    s3 = boto3.client('s3')

    # Rember to se stream = True.
    with contextlib.closing(requests.get(url, stream=True, verify=False)) as response:
        # Set up file stream from response content.
        fp = StringIO.StringIO(response.content)
        # Upload data to S3
        s3.upload_fileobj(fp, 'my-bucket', 'my-dir/' + url.split('/')[-1])