NodeJS JSON.stringify of very long array of objects errors “invalid string length”

问题:

I've got a very long array of objects that is about 100,000 items in size, and just before I got to write it to file, I pass the data into JSON.stringify

I get this error:

JSON.stringify(      ^ RangeError: Invalid string length 

How can I stringify a very large json object successfully?

Update:

since I initially had written this question, I've become more familiar with processing large amounts of data, and I prefer to use Apache Spark for this process. In Spark, it is customary to not use large JSON arrays, but rather a stream or file containing long strings of JSON, one object per line. A file would look like this:

{"name":"kristian", "favoriteSong":"enter sandman"} {"name":"joshua", "favoriteSong":"penny lane"} {"name":"susan", "favoriteSong":"all about that bass"} 

This is good because:

• you can arbitrarily deal with files that are terabytes large, and just as easily do some logicless file splitting and turn that into a thousand smaller files, were the logic is to simply break the file at some line break number N.
• This is better you wont ever have the potentially horrible issue of having a corrupted data file on your hands, caused by an un-closed JSON array.
• a large JSON array in a file tends to not have line breaks, and thus large files are a pain to open in VIM or other editors because it tries to load the whole thing into memory, as opposed to a file with linebreaks, where it can easily load the first 10 lines or so to the screen without issue.

回答1:

How can I stringify a very large json object successfully?

In parts? E.g., break the array up into smaller parts, use JSON.stringify on the smaller parts, and append each segment to the file (you'll have to do some [, ,, and ] handling).

回答2:

I find JSONStream to be a reliable alternative to the native JSON.stringify that works well with large objects. For example:

var fileSystem = require( "fs" ); var JSONStream = require( "JSONStream" ); var records = [     { id: 1, name: "Terminator" },     { id: 2, name: "Predator" },     { id: 3, name: "True Lies" },     { id: 4, name: "Running Man" },     { id: 5, name: "Twins" }     // .... hundreds of thousands of records .... ];  var transformStream = JSONStream.stringify(); var outputStream = fileSystem.createWriteStream( __dirname + "/data.json" ); transformStream.pipe( outputStream );     records.forEach( transformStream.write ); transformStream.end();  outputStream.on(     "finish",     function handleFinish() {         console.log("Done");     } ); 

Took the sample code from here.

文章来源: NodeJS JSON.stringify of very long array of objects errors “invalid string length”