Pass large amounts of data between web worker and main thread

Question

Is there a way to pass large amounts of data (multiple MB) between a web worker and the main thread? I work in a project where I need to download files, modify them a bit an

Answer 1

Firefox/Opera/Chrome all currently support a flavor of web workers called Transferable Objects which is extremely fast - also extremely easy to setup. Here we send to the ww (web worker) a browser allocated array which is populated by the ww and returned back to the browser side. This is passed by reference, not a copy : browser <-> ww

On the browser side :

var huge_array = new Float32Array(SIZE);

// worker.postMessage(huge_array.buffer);                      // old way
   worker.postMessage(huge_array.buffer, [huge_array.buffer]); // new Trans Obj

and then over inside the web worker :

self.onmessage = function(e) {

      var flt_arr = new Float32Array(e.data);

    //  typically you might want to populate flt_arr here

    //  now send data structure back to browser

    // self.postMessage(flt_arr.buffer);                    // old way
       self.postMessage(flt_arr.buffer, [flt_arr.buffer]); // new Trans Obj way
}

By simply putting the data object inside square brackets [in here] this tips off js to use Transferable Object mode. This also works when sending back and forth javascript objects containing multiple variables, not just typed arrays.

To quote :

Transferable objects are objects that are not copied (e.g. using something like structured cloning). Instead, the data is transferred from one context to another. The 'version' from the calling context is no longer available once transferred to the new context. For example, when transferring an ArrayBuffer from main app to Worker, the original ArrayBuffer from the main thread is cleared and no longer usable. This vastly improves performance of sending data to a Worker

http://html5-demos.appspot.com/static/workers/transferables/index.html https://html.spec.whatwg.org/multipage/workers.html

Answer 2

According to this WebWorkers tutorial, WebWorkers now support to pass File and Blob objects and basically any object that can be used with the structured clone algorith... or at least Chrome does it, probably because it implments the FileSystem API. I don't know if it's the main reason but I hope don't and in fact this feature is implemented in other browsers... being able to process user selected files in background is a nice thing.