Writing large files with Node.js

Question

I\'m writing a large file with node.js using a writable stream:

var fs     = require(\'fs\');
var stream = fs.createWriteStream(\'someFile.txt\', { flags : \         


        

Answer 1

The cleanest way to handle this is to make your line generator a readable stream - let's call it lineReader. Then the following would automatically handle the buffers and draining nicely for you:

lineReader.pipe(fs.createWriteStream('someFile.txt'));

If you don't want to make a readable stream, you can listen to write's output for buffer-fullness and respond like this:

var i = 0, n = lines.length;
function write () {
  if (i === n) return;  // A callback could go here to know when it's done.
  while (stream.write(lines[i++]) && i < n);
  stream.once('drain', write);
}
write();  // Initial call.

A longer example of this situation can be found here.