Large Base64 uploads in iOS

Question

Description: I need to send a base 64 encoded string in a HTTP POST parameter that represents a file to be uploaded.

Current method: I'm using a ASIFormDataRequest and encoding my file into a base64 string like so:

NSData *data = [[NSFileManager defaultManager] contentsAtPath:@"my path here"];
NSString *base64Data = [data encodeBase64ForData];

However when uploading large files the App runs out of memory and dies a horrid death!

Proposed solution: Does anybody now how I would go about, say, reading the file from disk, converting it to base64 strings in chunks and attaching the converted base64 string to a parameter in the HTTP request?

Effectively any way that avoids reading the whole file into memory.

Any input would be greatly appreciated!

My http request code in full:

NSURL *url = [NSURL URLWithString:requestProgressURLString];
NSData *data = [[NSFileManager defaultManager] contentsAtPath:evidence.uri];
NSString *base64Data = [data encodeBase64ForData];
NSURL *fileURL = [[NSURL alloc] initWithString:evidence.uri];

request = [ASIFormDataRequest requestWithURL:url];

request.shouldAttemptPersistentConnection = false;
[request setShowAccurateProgress:YES];

[request setPostValue:accessToken forKey:@"AccessToken"];
[request setPostValue:[[fileURL path] lastPathComponent] forKey:@"Filename"];
[request setPostValue:evidence.name forKey:@"Title"];
[request setPostValue:base64Data forKey:@"FileData"]; //wants string value (base64 encoded)          
[request addRequestHeader:@"Content-Type" value:@"application/x-www-form-urlencoded"];
[request setTimeOutSeconds:60];
[request setUploadProgressDelegate:self];
[request setDownloadProgressDelegate:self];
[request setDelegate:self];
[request setDidFinishSelector:@selector(requestDone:)];
[request setDidFailSelector:@selector(requestWentWrong:)];

Answer 1

I would split this problem into two:

1. Encode large file to Base64 in chunks
2. Upload that file

---

1. Encode large file to Base64 in chunks:

Find some code that encodes NSData instances to Base64. Then use one NSInputStream to read data in chunks, encode it and use NSOutputStream to write to a temporary file. I found this question/answer: Is it possible to base64-encode a file in chunks? Make the chunk size a multiple of 3 and it should work. You may want to move this to background thread.

Important: This will create a temporary file, that is larger than the original. There must be enough space on device for that moment. After you begin uploading you can delete it.

---

2. Upload that file

It seems that you have this already done. You can use that ASIFormDataRequest exactly like in your example. And since it makes a private copy of the file (it builds the multipart POST file), you can delete your copy.

If you want to optimize it by parallelizing these steps, you will need to find another way to upload the multipart file.

Answer 2

Instead of reading the file in one hit you need to read it in chunks

You can use an input stream to do this Apple have some info on here

Basically you need to create an NSInputStream pointed at your file as in the example

- (void)setUpStreamForFile:(NSString *)path {
    // iStream is NSInputStream instance variable
    iStream = [[NSInputStream alloc] initWithFileAtPath:path];
    [iStream setDelegate:self];
    [iStream scheduleInRunLoop:[NSRunLoop currentRunLoop]
        forMode:NSDefaultRunLoopMode];
    [iStream open];
}

You can then read the data in chunks using the NSStreamDelegate method - (void)stream:(NSStream *)stream handleEvent:(NSStreamEvent)eventCode

Again Apple give an example of this

- (void)stream:(NSStream *)stream handleEvent:(NSStreamEvent)eventCode {

    switch(eventCode) {
        case NSStreamEventHasBytesAvailable:
        {
            if(!_data) {
                _data = [[NSMutableData data] retain];
            }
            uint8_t buf[1024];
            unsigned int len = 0;
            len = [(NSInputStream *)stream read:buf maxLength:1024];
            if(len) {
                [_data appendBytes:(const void *)buf length:len];
                // bytesRead is an instance variable of type NSNumber.
                [bytesRead setIntValue:[bytesRead intValue]+len];

                // DO SOMETHING with DATA HERE

            } else {
                NSLog(@"no buffer!");
            }
            break;
        }

The size of chunks you choose will need to be a balance of memory performance vs network performance as there is obviously some overhead with the HTTP headers

You can also further optimise by sending your NSData directly ( and therefore save the base64 overhead) check this post

File Upload to HTTP server in iphone programming