Webworker-threads: is it OK to use “require” inside worker?

Question

(Using Sails.js)

I am testing webworker-threads ( https://www.npmjs.com/package/webworker-threads ) for long running processes on Node and the following example looks good:

var Worker = require('webworker-threads').Worker;
var fibo = new Worker(function() {
    function fibo (n) {
        return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
    }
    this.onmessage = function (event) {
        try{
            postMessage(fibo(event.data));                

        }catch (e){
            console.log(e);                
        }
    }
});
fibo.onmessage = function (event) {
    //my return callback
};
fibo.postMessage(40);

But as soon as I add any code to query Mongodb, it throws an exception: (not using the Sails model in the query, just to make sure the code could run on its own -- db has no password)

var Worker = require('webworker-threads').Worker;
var fibo = new Worker(function() {
    function fibo (n) {
        return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
    }
    // MY DB TEST -- THIS WORKS FINE OUTSIDE THE WORKER
    function callDb(event){
        var db = require('monk')('localhost/mydb');
        var users = db.get('users');
        users.find({ "firstName" : "John"}, function (err, docs){
            console.log(("serviceSuccess"));
            return fibo(event.data);
        });
    }
    this.onmessage = function (event) {
        try{

            postMessage(callDb(event.data));     // calling db function now           

        }catch (e){
            console.log(e);                
        }
    }
});
fibo.onmessage = function (event) {
    //my return callback
};
fibo.postMessage(40);

Since the DB code works perfectly fine outside the Worker, I think it has something to do with the require. I've tried something that also works outside the Worker, like

 var moment = require("moment");
 var deadline = moment().add(30, "s");

And the code also throws an exception. Unfortunately, console.log only shows this for all types of errors:

{Object}
{/Object}

So, the questions are: is there any restriction or guideline for using require inside a Worker? What could I be doing wrong here?

UPDATE

it seems Threads will not allow external modules https://github.com/xk/node-threads-a-gogo/issues/22

TL:DR I think that if you need to require, you should use a node's cluster or child process. If you want to offload some cpu busy work, you should use tagg and the load function to grab any helpers you need.

Upon reading this thread, I see that this question is similar to this one: Load Nodejs Module into A Web Worker

To which Audreyt, the webworker-threads author answered:

author of webworker-threads here. Thank you for using the module!

There is a default native_fs_ object with the readFileSync you can use to read files.

Beyond that, I've mostly relied on onejs to compile all required modules in package.json into a single JS file for importScripts to use, just like one would do when deploying to a client-side web worker environment. (There are also many alternatives to onejs -- browserify, etc.)

Hope this helps!

So it seems importScripts is the way to go. But at this point, it might be too hacky for what I want to do, so probably KUE is a more mature solution.

Answer 1

I'm a collaborator on the node-webworker-threads project.

You can't require in node-webworker-threads

You are correct in your update: node-webworker-threads does not (currently) support requireing external modules.

It has limited support for some of the built-ins, including file system calls and a version of console.log. As you've found, the version of console.log implemented in node-webworker-threads is not identical to the built-in console.log in Node.js; it does not, for example, automatically make nice string representations of the components of an Object.

In some cases you can use external modules, as outlined by audreyt in her response. Clearly this is not ideal, and I view the incomplete require as the primary "dealbreaker" of node-webworker-threads. I'm hoping to work on it this summer.

When to use node-webworker-threads

node-webworker-threads allows you to code against the WebWorker API and run the same code in the client (browser) and the server (Node.js). This is why you would use node-webworker-threads over node-threads-a-gogo.

node-webworker-threads is great if you want the most lightweight possible JavaScript-based workers, to do something CPU-bound. Examples: prime numbers, Fibonacci, a Monte Carlo simulation, offloading built-in but potentially-expensive operations like regular expression matching.

When not to use node-webworker-threads

node-webworker-threads emphasizes portability over convenience. For a Node.js-only solution, this means that node-webworker-threads is not the way to go.

If you're willing to compromise on full-stack portability, there are two ways to go: speed and convenience.

For speed, try a C++ add-on. Use NaN. I recommend Scott Frees's C++ and Node.js Integration book to learn how to do this, it'll save you a lot of time. You'll pay for it in needing to brush up on your C++ skills, and if you want to work with MongoDB then this probably isn't a good idea.

For convenience, use a Child Process-based worker pool like fork-pool. In this case, each worker is a full-fledged Node.js instance. You can then require to your heart's content. You'll pay for it in a larger application footprint and in higher communication costs compared to node-webworker-threads or a C++ add-on.