Streaming file data into mongodb gridfs

问题

I'm trying to upload video files to gridfs using django + mongoengine on server.

Client Side: (JavaScript to read/chunk the file and send the data to the server using ajax. )

_upload : function() {
    chunk = self.file.slice( self.start, self.end );
    reader = new FileReader();
    reader.readAsDataURL( chunk );
    reader.onload = function(e) {
        this.request = new XMLHttpRequest();
        this.request.open( 'POST', '/ajax/video_upload/' );
        this.request.setRequestHeader("X-CSRFToken", getCookie('csrftoken'));
        this.request.overrideMimeType('application/octet-stream');
        this.request.send( JSON.stringify({ 'chunk': e.target.result, 'id' : self.file_id }) );
        this.request.onload = function() {

        if( self.start >= self.file_size && self.preventedOverflow ) {
        return;
        }

        self.start = self.end;
        self.end = self.end + self.chunkSize;

       self._upload();
    };
}

Server Side:

def uploadVideo(request):
if request.body and request.is_ajax:
    data = json.loads(request.body)
    m = Multimedia.objects.get( id = data['id'] )
    m.media.new_file()
    m.media.write( data['chunk'] )
    m.media.close()
    m.save()
    return HttpResponse()

Error:

ERROR:django.request:Internal Server Error: /ajax/video_upload/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/praveen/Desktop/gatherify/gatherify/../ajax/views.py", line 33, in uploadVideo
    m.media.write( data['chunk'] )
  File "/usr/local/lib/python2.7/dist-packages/mongoengine-0.8.7-py2.7.egg/mongoengine/fields.py", line 1172, in write
    self.newfile.write(string)
  File "build/bdist.linux-i686/egg/gridfs/grid_file.py", line 327, in write
    "order to write %s" % (text_type.__name__,))
TypeError: must specify an encoding for file in order to write unicode

I have no idea how to specify the encoding the official documentation doesn't say anything about it. (http://mongoengine-odm.readthedocs.org/guide/gridfs.html)

Another problem is that when I try to write the next chunks on the next ajax request I get an error:

GridFSError: This document already has a file. Either delete it or call replace to overwrite it

Any help is appreciated. Thanks :)

回答1:

Encode the data['chunk'] string before writing it to the FileField.

m.media.new_file()
m.media.write( data['chunk'].encode("UTF-8") )
m.media.close()

As for your second question you've already created a file in gridfs. Like the error message says you've got to m.media.delete() it or m.media.replace(<a new gridfs entry>) it. If you're looking to append it you're probably going to have to m.media.get() the file contents as a string, append the new chunk to the string, then create a new m.media gridfs file. You can't edit gridfs content directly.

回答2:

1. you need to write the data in utf-8
2. you shouldn't close the GridOut instance obtained from newfile after you have written just the first chunk
3. you should create a greenlet for each new file upload
4. yield after writing the chunk
5. send ack to receive next chunk also some 'id' to identify the greenlet.
6. wake up greenlet and send the new chunk
7. send 'end of file' once no chunks are left
8. close the GridOut now.
9. exit the greenlet

来源：https://stackoverflow.com/questions/24270876/streaming-file-data-into-mongodb-gridfs