Python script to use data from Azure Storage Blob by stream, and update blob by stream without local file reading and uploading

问题

I have a python code for data processing , i want to use azure block blob as the data input for the code, to be specify, a csv file from block blob. its all good to download the csv file from azure blob to local path, and upload other way around for the python code if running locally, but the problem is my code running on azure virtual machine because its quite heavy for my Apple Air , pandas read_csv from local path does not work in this case, therefore i have to download and upload and update csv files to azure storage by stream without local saving. both download and upload csv are quite small volume, much less than the blob block limits

there wasn't that much tutorial to explain how to do this step by step, MS Docs are generally suck to explain as well, my minimal code are as following:

for downloading from azure blob storage

from azure.storage.blob import BlockBlobService
storage = BlockBlobService(account_name='myname', account_key = 'mykey')
#here i don't know how to make a csv stream that could could be used in next steps#
file = storage.get_blob_to_stream('accountname','blobname','stream')
df = pd.read_csv(file)
#df for later steps#

for uploading and updating a blob by a dataframe from code by stream

df #dataframe generated by code 
'i don't know how to do the preparation steps for df and the final fire up operation'
storage.put_blob_to_list _by_stream('accountname','blobname','stream')

can you please make a step by step tutorial for me, for ppl has experience to azure blob , this should be not very difficult.

or if you have better solution other than use blob for my case , please drop some hits. Thanks.

回答1:

So the document is still in progress, I think it is getting better and better... Useful link:

Github - Microsoft Azure Storage SDK for Python
Quickstart: Upload, download, and list blobs using Python

To download a file as a stream from blob storage, you can use BytesIO:

from azure.storage.blob import BlockBlobService
from io import BytesIO
from shutil import copyfileobj 
with BytesIO() as input_blob:
    with BytesIO() as output_blob:
        block_blob_service = BlockBlobService(account_name='my_account_name', account_key='my_account_key')
        # Download as a stream
        block_blob_service.get_blob_to_stream('mycontainer', 'myinputfilename', input_blob)

        # Do whatever you want to do - here I am just copying the input stream to the output stream
        copyfileobj(input_blob, output_blob)
        ...

        # Create the a new blob
        block_blob_service.create_blob_from_stream('mycontainer', 'myoutputfilename', output_blob)

        # Or update the same blob
        block_blob_service.create_blob_from_stream('mycontainer', 'myinputfilename', output_blob)

来源：https://stackoverflow.com/questions/49467961/python-script-to-use-data-from-azure-storage-blob-by-stream-and-update-blob-by

标签

python

azure

azure-blob-storage