Python bottle - How to upload media files without DOSing the server

Question

I was using the answer from this question and saw the comment:

   raw = data.file.read() # This is dangerous for big files

How do I upload the file without doing this? My code so far is:

@bottle.route('/uploadLO', method='POST')
def upload_lo():
    upload_dir = get_upload_dir_path()
    files = bottle.request.files
    print files, type(files)
    if(files is not None):
        file = files.file
        print file.filename, type(file)
        target_path = get_next_file_name(os.path.join(upload_dir, file.filename))
        print target_path
        shutil.copy2(file.read(), target_path)  #does not work. Tried it as a replacement for php's move_uploaded_file
    return None

which gives this output:

127.0.0.1 - - [03/Apr/2014 09:29:37] "POST /uploadLO HTTP/1.1" 500 1418
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\bottle.py", line 862, in _handle
    return route.call(**args)
  File "C:\Python27\lib\site-packages\bottle.py", line 1727, in wrapper
    rv = callback(*a, **ka)
  File "C:\dev\project\src\mappings.py", line 83, in upload_lo
    shutil.copy2(file.read(), target_path)
AttributeError: 'FileUpload' object has no attribute 'read'

I am using python bottle v.12, dropzone.min.js, and mongodb. I also am using this tutorial:

http://www.startutorial.com/articles/view/how-to-build-a-file-upload-form-using-dropzonejs-and-php

Answer 1

This is called "file slurping":

raw = data.file.read() 

and you don't want to do it (at least in this case).

Here's a better way to read a binary file of unknown (possibly large) size:

data_blocks = []

buf = data.file.read(8192)
while buf:
    data_blocks.append(buf)
    buf = data.file.read(8192)

data = ''.join(data_blocks)

You may also want to stop iterating if the accumulated size exceeds some threshold.

Hope that helps!

---

PART 2

You asked about limiting the file size, so here's an amended version that does that:

MAX_SIZE = 10 * 1024 * 1024 # 10MB
BUF_SIZE = 8192

# code assumes that MAX_SIZE >= BUF_SIZE

data_blocks = []
byte_count = 0

buf = f.read(BUF_SIZE)
while buf:
    byte_count += len(buf)

    if byte_count > MAX_SIZE:
        # if you want to just truncate at (approximately) MAX_SIZE bytes:
        break
        # or, if you want to abort the call
        raise bottle.HTTPError(413, 'Request entity too large (max: {} bytes)'.format(MAX_SIZE))

    data_blocks.append(buf)
    buf = f.read(BUF_SIZE)

data = ''.join(data_blocks)

It's not perfect, but it's simple and IMO good enough.

Answer 2

To add to ron.rothman's excellent answer...to fix your error message try this

@bottle.route('/uploadLO', method='POST')
def upload_lo():
    upload_dir = get_upload_dir_path()
    files = bottle.request.files

    # add this line
    data = request.files.data

    print files, type(files)

    if(files is not None):
        file = files.file
        print file.filename, type(file)
        target_path = get_next_file_name(os.path.join(upload_dir, file.filename))
        print target_path

        # add Ron.Rothman's code
        data_blocks = []
        buf = data.file.read(8192)
        while buf:
            data_blocks.append(buf)
            buf = data.file.read(8192)

        my_file_data = ''.join(data_blocks)
        # do something with the file data, like write it to target
        file(target_path,'wb').write(my_file_data)

    return None