Understanding Node JS Generators with fs Module

Question

I\'ve been very excited about Node JS for awhile. I finally decided to knuckle down and write a test project to learn about generators in the latest Harmony build of Node.

Answer 1

@loganfsmyth already provides a great answer to your question. The goal of my answer is to help you understand how JavaScript generators actually work, as this is a very important step to using them correctly.

Generators implement a state machine, the concept which is nothing new by itself. What's new is that generators allow to use the familiar JavaScript language construct (e.g., for, if, try/catch) to implement a state machine without giving up the linear code flow.

The original goal for generators is to generate a sequence of data, which has nothing to do with asynchrony. Example:

// with generator

function* sequence()
{
    var i = 0;
    while (i < 10)
        yield ++i * 2;
}

for (var j of sequence())
    console.log(j);

// without generator

function bulkySequence()
{
    var i = 0;
    var nextStep = function() {
        if ( i >= 10 )
            return { value: undefined, done: true };
        return { value: ++i * 2, done: false };
    }
    return { next: nextStep };
}

for (var j of bulkySequence())
    console.log(j);

The second part (bulkySequence) shows how to implement the same state machine in the traditional way, without generators. In this case, we no longer able to use while loop to generate values, and the continuation happens via nextStep callback. This code is bulky and unreadable.

Let's introduce asynchrony. In this case, the continuation to the next step of the state machine will be driven not by for of loop, but by some external event. I'll use a timer interval as a source of the event, but it may as well be a Node.js operation completion callback, or a promise resolution callback.

The idea is to show how it works without using any external libraries (like Q, Bluebird, Co etc). Nothing stops the generator from self-driving itself to the next step, and that's what the following code does. Once all steps of the asynchronous logic have completed (the 10 timer ticks), doneCallback will be invoked. Note, I don't return any meaningful data with yield here. I merely use it to suspend and resume the execution:

function workAsync(doneCallback)
{
    var worker = (function* () {
        // the timer callback drivers to the next step
        var interval = setInterval(function() { 
            worker.next(); }, 500);

        try {
            var tick = 0;
            while (tick < 10 ) {
                // resume upon next tick
                yield null;
                console.log("tick: " + tick++);
            }
            doneCallback(null, null);
        }
        catch (ex) {
            doneCallback(ex, null);
        }
        finally {
            clearInterval(interval);
        }
    })();

    // initial step
    worker.next();
}

workAsync(function(err, result) { 
    console.log("Done, any errror: " + err); });

Finally, let's create a sequence of events:

function workAsync(doneCallback)
{
    var worker = (function* () {
        // the timer callback drivers to the next step
        setTimeout(function() { 
            worker.next(); }, 1000);

        yield null;
        console.log("timer1 fired.");

        setTimeout(function() { 
            worker.next(); }, 2000);

        yield null;
        console.log("timer2 fired.");

        setTimeout(function() { 
            worker.next(); }, 3000);

        yield null;
        console.log("timer3 fired.");

        doneCallback(null, null);
    })();

    // initial step
    worker.next();
}

workAsync(function(err, result) { 
    console.log("Done, any errror: " + err); });

Once you understand this concept, you can move on with using promises as wrappers for generators, which takes it to the next powerful level.