How do I read the contents of a Node.js stream into a string variable?

Question

I\'m hacking on a Node program that uses smtp-protocol to capture SMTP emails and act on the mail data. The library provides the mail data as a stream, and I don\'t know how

Answer 1

(This answer is from years ago, when it was the best answer. There is now a better answer below this. I haven't kept up with node.js, and I cannot delete this answer because it is marked "correct on this question". If you are thinking of down clicking, what do you want me to do?)

The key is to use the data and end events of a Readable Stream. Listen to these events:

stream.on('data', (chunk) => { ... });
stream.on('end', () => { ... });

When you receive the data event, add the new chunk of data to a Buffer created to collect the data.

When you receive the end event, convert the completed Buffer into a string, if necessary. Then do what you need to do with it.

Answer 2

And yet another one for strings using promises:

function getStream(stream) {
  return new Promise(resolve => {
    const chunks = [];

    # Buffer.from is required if chunk is a String, see comments
    stream.on("data", chunk => chunks.push(Buffer.from(chunk)));
    stream.on("end", () => resolve(Buffer.concat(chunks).toString()));
  });
}

Usage:

const stream = fs.createReadStream(__filename);
getStream(stream).then(r=>console.log(r));

remove the .toString() to use with binary Data if required.

update: @AndreiLED correctly pointed out this has problems with strings. I couldn't get a stream returning strings with the version of node I have, but the api notes this is possible.

Answer 3

What do you think about this ?

// lets a ReadableStream under stream variable 
const chunks = [];

for await (let chunk of stream) {
    chunks.push(chunk)
}

const buffer  = Buffer.concat(chunks);
const str = buffer.toString("utf-8")

Answer 4

Streams don't have a simple .toString() function (which I understand) nor something like a .toStringAsync(cb) function (which I don't understand).

So I created my own helper function:

var streamToString = function(stream, callback) {
  var str = '';
  stream.on('data', function(chunk) {
    str += chunk;
  });
  stream.on('end', function() {
    callback(str);
  });
}

// how to use:
streamToString(myStream, function(myStr) {
  console.log(myStr);
});

Answer 5

All the answers listed appear to open the Readable Stream in flowing mode which is not the default in NodeJS and can have limitations since it lacks backpressure support that NodeJS provides in Paused Readable Stream Mode. Here is an implementation using Just Buffers, Native Stream and Native Stream Transforms and support for Object Mode

import {Transform} from 'stream';

let buffer =null;    

function objectifyStream() {
    return new Transform({
        objectMode: true,
        transform: function(chunk, encoding, next) {

            if (!buffer) {
                buffer = Buffer.from([...chunk]);
            } else {
                buffer = Buffer.from([...buffer, ...chunk]);
            }
            next(null, buffer);
        }
    });
}

process.stdin.pipe(objectifyStream()).process.stdout

Answer 6

Hope this is more useful than the above answer:

var string = '';
stream.on('data',function(data){
  string += data.toString();
  console.log('stream data ' + part);
});

stream.on('end',function(){
  console.log('final output ' + string);
});

Note that string concatenation is not the most efficient way to collect the string parts, but it is used for simplicity (and perhaps your code does not care about efficiency).

Also, this code may produce unpredictable failures for non-ASCII text (it assumes that every character fits in a byte), but perhaps you do not care about that, either.