Handling large file uploads with Flask

Question

What would be the best way to handle very large file uploads (1 GB +) with Flask?

My application essentially takes multiple files assigns them one unique file number

Answer 1

When uploading a file, you just can't leave the page and have it continue. The page has to stay opened in order to continue the upload.

Something you could do is open a new tab just for handling the upload and alerting the user when they inadvertently close the new tab before the upload finishes. That way the upload will be separate from whatever the user is doing on the original page so they can still navigate without cancelling the upload. The upload tab can also just close itself when it finishes.

index.js

    // get value from  on page
    var upload = document.getElementById('upload');
    upload.addEventListener('input', function () {
        // open new tab and stick the selected file in it
        var file = upload.files[0];
        var uploadTab = window.open('/upload-page', '_blank');
        if (uploadTab) {
            uploadTab.file = file;
        } else {
            alert('Failed to open new tab');
        }
    });

upload-page.js

    window.addEventListener('beforeunload', function () {
        return 'The upload will cancel if you leave the page, continue?';
    });
    window.addEventListener('load', function () {
        var req = new XMLHttpRequest();
        req.addEventListener('progress', function (evt) {
            var percentage = '' + (evt.loaded / evt.total * 100) + '%';
            // use percentage to update progress bar or something
        });
        req.addEventListener('load', function () {
            alert('Upload Finished');
            window.removeEventListener('beforeunload');
            window.close();
        });
        req.addRequestHeader('Content-Type', 'application/octet-stream');
        req.open('POST', '/upload/'+encodeURIComponent(window.file.name));
        req.send(window.file);
    });

On the server, you can use request.stream to read the uploaded file in chunks to avoid having to wait for the entire thing to load up in memory first.

server.py

@app('/upload/', methods=['POST'])
def upload(filename):
    filename = urllib.parse.unquote(filename)
    bytes_left = int(request.headers.get('content-length'))
    with open(os.path.join('uploads', filename), 'wb') as upload:
        chunk_size = 5120
        while bytes_left > 0:
            chunk = request.stream.read(chunk_size)
            upload.write(chunk)
            bytes_left -= len(chunk)
        return make_response('Upload Complete', 200)

You might be able to use FormData api instead of an octet-stream, but I'm not sure if you can stream those in flask.