Uploading Large NSData to the Web

Question

I\'m currently working on an application that has to upload large files (mainly movies/videos) to the web. After reading what I can, I went the the approach of converting t

Answer 1

Assuming that it makes sense to upload the movie in its native format, you can really make this easier using the BSD (ie Unix) section 3 interface:

• given a filePath, open the file and get an int file descriptor (fd)

• with fd, get the length of the file

• keep track of how much you've loaded so you know where to get more data

• use mmap(3) to map in JUST the data you want to upload at any time, and use the void * pointer returned by mmap as the location of the data

• when the data has been sent, munmap the old data chunk and mmap a new chunk

• after all data is sent, munmap the last chunk, the close(fd).

No temporary memory - no mallocs. I use mmap whenever I have to deal with huge files.

Edit: you can also use NSData dataWithContentsOfFile:options with options set to use mmap. You would then use the byte pointer to read small chunks as you need them.

Answer 2

In case anyone got here and couldn't solve your problems, I figured out a way to do this. You have to firstly write your ALAssetRepresentation to disk (as described here):

NSUInteger chunkSize = 100 * 1024;
NSString *tempFile = [NSTemporaryDirectory() stringByAppendingPathComponent:@"temp.tmp"];

uint8_t *chunkBuffer = malloc(chunkSize * sizeof(uint8_t));
NSUInteger length = [rep size];

NSFileHandle *fileHandle = [[NSFileHandle fileHandleForWritingAtPath: tempFile] retain];
if(fileHandle == nil) {
    [[NSFileManager defaultManager] createFileAtPath:tempFile contents:nil attributes:nil];
    fileHandle = [[NSFileHandle fileHandleForWritingAtPath:tempFile] retain];
}

NSUInteger offset = 0;
do {
    NSUInteger bytesCopied = [rep getBytes:chunkBuffer fromOffset:offset length:chunkSize error:nil];
    offset += bytesCopied;
    NSData *data = [[NSData alloc] initWithBytes:chunkBuffer length:bytesCopied];
    [fileHandle writeData:data];
    [data release];
} while (offset < length);
[fileHandle closeFile];
[fileHandle release];
free(chunkBuffer);
chunkBuffer = NULL;

Then you have to create an NSData object that can map the disk without using memory resources (kind of like David's answer, but inspired by this answer):

NSError *error;
NSData *fileData = [NSData dataWithContentsOfFile:tempFile options:NSDataReadingMappedIfSafe error:&error];
if (!fileData) {
    NSLog(@"Error %@ %@", error, [error description]);
    NSLog(@"%@", tempFile);
    //do what you need with the error
}

EDIT Although, if you are uploading the file somewhere, you should open a connection and send small buffers of the file, kind of like what I did above. I had to write a C++ class to handle the socket and the connection

Answer 3

You probably shouldn't be trying to read the whole asset in one shot:

Byte *buffer = (Byte *)malloc([representation size]);
NSUInteger buffered = [representation getBytes:buffer fromOffset:0.0 length:[representation size] error:nil];

Instead, set up a loop and read from the asset in chunks. I've outlined the basic approach. You'll need to fill in a few gaps, but it should solve the memory issue.

You might also want to consider running this in a thread so you don't lock up the UI.

    NSError error;
    int bufferSize = 1000;
    float offset=0.0;

//TODO: Open Connection

    while (1)
    {
      Byte *buffer = (Byte *)malloc(bufferSize);
       NSUInteger buffered = [representation getBytes:buffer fromOffset:offset length:bufferSize  error:&error];

    //TODO: Write data  
    //TODO: Increment offset, check errors

    free(buffer);

    //if (done){
    //break;
    //}

    }

    //TODO close eonnection