File compression before upload on the client-side

Question

Basically I\'ll be working with large XML files (approx. 20 - 50 MB). These files needs to be uploaded on a server.

I know it isn\'t possible to touch the files with

Answer 1

With Silverlight, you can zip files on the client side, and this approach works in all major browsers. Moreover, you can interact with your Silverlight widget via JavaScript. Also, if a user needs to upload several files, your Silverlight widget can show a single dialog for selecting all the files. The only drawback is that your clients must install the Silverlight plugin.

Answer 2

If for whatever reason you can't get a solution to work in JavaScript for all major browsers, I know of an AS3 compression library here: http://code.google.com/p/ascompress/.

Also, a less cooler option, if your target users are somewhat tech savy why not have them upload a .zip file of the xml? Then on the server side you can unzip and process as needed.

Either way on the server side you'll want to unzip/decompress, which should be easy to google for solutions if you don't already have one in mind.

Answer 3

Flash's inbuilt implementation of ByteArray has a method (ByteArray::deflate to deflate the contents (of the bytearray) The deflate algorithm is the DEFLATE Compressed Data Format Specification version 1.3.

There;s also a ByteArray::compress method which compresses using the zlib algorithm

Hold on a bit, I'll write you some sample code to use this class and expose it to JavaScript.

EDIT

I've uploaded the file at http://www.filefactory.com/file/cf8a39c/n/demo5.zip

EDIT 2 For those who couldn't download the files:

My ActionScript code in demo5.fla (compiled to demo5.swf)

import flash.external.ExternalInterface;
import flash.net.FileReference;
import flash.events.Event;
import flash.utils.ByteArray;

if(ExternalInterface.available) {
    //flash.system.Security.allowDomain("localhost");
    ExternalInterface.addCallback("deflate", doDeflate);
    ExternalInterface.addCallback("compress", doCompress);
}

var method:String="deflate";
var b:ByteArray;
function doCompress(_data:String):void {
    method="compress";
    exec(_data);
}

function doDeflate(_data:String):void {
    method="deflate";
    exec(_data);
}

function exec(_data:String):void {
    b=new ByteArray();
    b.writeUTFBytes(_data);
    b.position=0;
    if(method=="compress") {
        b.compress();
    } else if(method=="deflate") {
        b.deflate();
    }
    executed();
}

function executed():void {
    if(ExternalInterface.available) {
        b.position=0;
        var str:String=b.readUTFBytes(b.bytesAvailable);
        ExternalInterface.call("onExec", str);
    }
}

My HTML code to embed the swf:

<button onclick="doDeflate()">Deflate</button>
<button onclick="doCompress()">Compress</button>
<div id="flashContent">
    <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" width="1" height="1" id="demo5" align="middle">
        <param name="movie" value="demo5.swf" />
        <param name="quality" value="high" />
        <param name="bgcolor" value="#ffffff" />
        <param name="play" value="true" />
        <param name="loop" value="true" />
        <param name="wmode" value="window" />
        <param name="scale" value="showall" />
        <param name="menu" value="true" />
        <param name="devicefont" value="false" />
        <param name="salign" value="" />
        <param name="allowScriptAccess" value="always" />

        <embed src="demo5.swf" quality="high" bgcolor="#869ca7"
             width="1" height="1" name="demo5" align="middle"
             play="true" loop="false" quality="high" allowScriptAccess="always"
             type="application/x-shockwave-flash"
             pluginspage="http://www.macromedia.com/go/getflashplayer">
        </embed>
    </object>
</div>

and finally the javascript code:

function doDeflate() {
    var data="fdg fhnkl,hgltrebdkjlgyu ia43uwriu67ri8m nirugklhvjsd fgvu";
    //DATA CONTAINS DATA TO BE DEFLATED
    thisMovie("demo5").deflate(data);
}

function doCompress() {
    var data="fdg fhnkl,hgltrebdkjlgyu ia43uwriu67ri8m nirugklhvjsd fgvu";
    //DATA CONTAINS DATA TO BE DEFLATED
    thisMovie("demo5").compress(data);
}

function onExec(data) {
    //DATA CONTAINS THE DEFLATED DATA
    alert(data);
}

function thisMovie(movieName) {
    if (navigator.appName.indexOf("Microsoft") != -1) {
        return window[movieName];
    } else {
        return document[movieName];
    }
}

Answer 4

You can make use of JSZip. For input, it supports String/ArrayBuffer/Uint8Array/Buffer, but not blobs, which is what you get from an <input type="file"/> with javascript:

A File object is specific kind of a Blob, and can be used in any context that a Blob can

(link)

So you'll have to convert the blob/file to e.g. an ArrayBuffer first, e.g. using FileReader.readAsArrayBuffer(). Note that this function works asynchronously, demanding callback usage. There is also a FileReaderSync available, yet "This interface is only available in workers as it enables synchronous I/O that could potentially block", so I don't see any good in using it.

(EDIT. I'm not sure but I believe you can skip the blob->ArrayBuffer conversion now and simply zip the File object.)

This whole approach is specially useful if php's directive max_file_uploads was set to a small number by your webspace host, for now the only thing you'll have to worry about is upload_max_filesize

For reference, a code sample excerpt follows (using JQuery) for putting several files of one multiple file input in a zip before submitting:

// onclick:
var fileInput = $(':file');
var files = [];
$.each(fileInput[0].files, function(i, file) {
    files.push(file);
});

var zip = new JSZip();
function addFileToZip(n) {
    if(n >= files.length) {
        zippingComplete(zip.generate({type:"blob", compression:"deflate"}));
        return;
    }
    var file = files[n];                    
    var arrayBuffer;
    var fileReader = new FileReader();
    fileReader.onload = function() {
        arrayBuffer = this.result;
        zip.file(file.name, arrayBuffer);
        addFileToZip(n + 1);
    };
    fileReader.readAsArrayBuffer(file);
}
addFileToZip(0);

function zippingComplete(zip) {
    formData = new FormData();
    formData.append('fileZip', zip);
    formData.append("param1", "blah");
    $.ajax({
        data: formData,
        //... etc

Server-side-wise, you'll access $_FILES["fileZip"].

Answer 5

There some javascript library of huffman compression freely available, for example https://github.com/wilkerlucio/huffman_js but I think that your task is impossible because with javascript and html it's not possible to load huge data into the browser or client's memory.

Answer 6

Consider reviewing this other stackoverflow post. Reading both answers paints a good picture of compression reality.

I'm considering implementing a Silverlight of Flex solution that compresses client side and if the user doesn't want to install it, compress and decompress the file server side. Will update this post when a solution is found.

Installing the control would be sold to the user as a time saver, which is normally true. For the server, it would be a bandwidth and compression processing saver.