Is there a way to stream data directly from python request to minio bucket

Question

I am trying to make a GET request to a server to retrieve a tiff image. I then want to stream it directly to MinIO using the put_object method in the MinIO python SDK.

Answer 1

You can stream your file directly into a minio bucket like this:

import requests

from pathlib import Path
from urllib.parse import urlparse

from django.conf import settings
from django.core.files.storage import default_storage

client = default_storage.client
object_name = Path(urlparse(response.url).path).name
bucket_name = settings.MINIO_STORAGE_MEDIA_BUCKET_NAME

with requests.get(url_to_download, stream=True) as r:
    content_length = int(r.headers["Content-Length"])
    result = client.put_object(bucket_name, object_name, r.raw, content_length)

Or you can use a django file field directly:

with requests.get(url_to_download, stream=True) as r:
    # patch the stream to make django-minio-storage belief
    # it's about to read from a legit file
    r.raw.seek = lambda x: 0
    r.raw.size = int(r.headers["Content-Length"])
    model = MyModel()
    model.file.save(object_name, r.raw, save=True)

The RawIOBase hint from Dinko Pehar was really helpful, thanks a lot. But you have to use response.raw not response.content which would download your file immediately and be really inconvenient when trying to store a large video for example.