Trying to understand generators / yield in node.js - what executes the asynchronous function?

Question

Node.js now has generators.

My understanding is that generators can be used to write code that appears to be much more linear and avoids callback hell and pyramid of doom style coding.

So to this point, my understanding is that inside a generator, code executes until it reaches a "yield" statement. Execution of the generator function suspends at this point. The yield statement specifies a return value which may be a function. Typically this would be a blocking I/O function - one that would normally need to be executed asynchronously.

The yield's return function is returned to whatever called the generator.

My question is, what happens at this point? What exactly executes the blocking I/O function that the yield returned?

Is it correct that to write generator/yield code that appears to be linear, there needs to be a specific sort of function that is calling the generator, a function that loops through the generator and executes each asynch function returned by the yield and returns the result of the asynch function back into the generator?

It's still not clear to me exactly how the asynch function returned by the yield is executed. If it is executed by the function that calls the generator, is it executed asynchronously? I'm guessing so because to do otherwise would result in blocking behaviour.

To summarise my questions:

1. To write "linear" asynch code with generators, is it necessary for there to be a calling function that iterates over the generator, executing yielded functions as callbacks and returning the result of the callback back into the generator?
2. If the answer to question 1 is yes, the exactly how are the yielded functions executed - asynchronously?

Can anyone offer a better overview/summary of how the whole process works?

Answer 1

When writing async code with generators you are dealing with two types of functions:

• normal functions declared with function. These functions cannot yield. You cannot write async code in sync style with them because they run to completion; you can only handle asynchronous completion through callbacks (unless you invoke extra power like the node-fibers library or a code transform).
• generator functions declared with function*. These functions can yield. You can write async code in sync style inside them because they can yield. But you need a companion function that creates the generator, handles the callbacks and resumes the generator with a next call every time a callback fires.

There are several libraries that implement companion functions. In most of these libraries, the companion function handles a single function* at a time and you have to put a wrapper around every function* in your code. The galaxy library (that I wrote) is a bit special because it can handle function* calling other function* without intermediate wrappers. The companion function is a bit tricky because it has to deal with a stack of generators.

The execution flow can be difficult to understand because of the little yield/next dance between your function* and the companion function. One way to understand the flow is to write an example with the library of your choice, add console.log statements both in your code and in the library, and run it.

Answer 2

If [the blocking io function] is executed by the function that calls the generator, is it executed asynchronously? I'm guessing so because to do otherwise would result in blocking behaviour.

I don't think generators do asynchronous task handling. With generators, only one thing is executing at the same time--it's just that one function can stop executing and pass control to another function. For instance,

function iofunc1() {
  console.log('iofunc1');
}

function iofunc2() {
  console.log('iofunc2');
}

function* do_stuff() {
  yield iofunc1;
  yield iofunc2;
  console.log('goodbye');
}


var gen = do_stuff();
(gen.next().value)(); 
(gen.next().value)(); //This line won't begin execution until the function call on the previous line returns
gen.next(); //continue executing do_stuff

If you read some of the articles about nodejs generators:

1. http://jlongster.com/2012/10/05/javascript-yield.html
2. http://jlongster.com/A-Study-on-Solving-Callbacks-with-JavaScript-Generators
3. http://jlongster.com/A-Closer-Look-at-Generators-Without-Promises

...they all employ additional functions/libraries to add in asynchronous execution.

Answer 3

1: to write "linear" asynch code with generators, is it necessary for there to be a calling function that iterates over the generator, executing yielded functions as callbacks and returning the result of the callback back into the generator?

Yes. Let's call it "launcher".

2: if the answer to question 1 is yes, the exactly how are the yielded functions executed - asynchronously?

Inside the generator, you yield an array with: the function and its parameters. In the controlling caller (launcher), you use fn.apply(..,callback) to call the async, putting the call to "generator.next(data);" (resume) inside the callback.

the async function is executed asynchronously, but the generator will be "paused" at the yield point, until the callback is called (and then "generator.next(data)" is executed)

Full working lib and samples: https://github.com/luciotato/waitfor-es6