Node.js heap out of memory

问题:

Today I ran my script for filesystem indexing to refresh RAID files index and after 4h it crashed with following error:

[md5:]  241613/241627 97.5%   [md5:]  241614/241627 97.5%   [md5:]  241625/241627 98.1% Creating missing list... (79570 files missing) Creating new files list... (241627 new files)    11629672 ms: Mark-sweep 1174.6 (1426.5) -> 1172.4 (1418.3) MB, 659.9 / 0 ms [allocation failure] [GC in old space requested]. 11630371 ms: Mark-sweep 1172.4 (1418.3) -> 1172.4 (1411.3) MB, 698.9 / 0 ms [allocation failure] [GC in old space requested]. 11631105 ms: Mark-sweep 1172.4 (1411.3) -> 1172.4 (1389.3) MB, 733.5 / 0 ms [last resort gc]. 11631778 ms: Mark-sweep 1172.4 (1389.3) -> 1172.4 (1368.3) MB, 673.6 / 0 ms [last resort gc].     ==== JS stack trace =========================================  Security context: 0x3d1d329c9e59  1: SparseJoinWithSeparatorJS(aka SparseJoinWithSeparatorJS) [native array.js:~84] [pc=0x3629ef689ad0] (this=0x3d1d32904189 ,w=0x2b690ce91071 ,L=241627,M=0x3d1d329b4a11 ,N=0x7c953bf4d49 ) 2: Join(aka Join) [native array.js:143] [pc=0x3629ef616696] (this=0x3d1d32904189 

Server is equipped with 16gb RAM and 24gb SSD swap. I highly doubt my script exceeded 36gb of memory. At least it shouldn't

Script creates index of files stored as Array of Objects with files metadata (modification dates, permissions, etc, no big data)

Here's full script code: http://pastebin.com/mjaD76c3

I've already experiend weird node issues in the past with this script what forced me eg. split index into multiple files as node was glitching when working on such big files as String. Is there any way to improve nodejs memory management with huge datasets?

回答1:

If I remember correctly, there is a strict standard limit for the memory usage in V8 of around 1.7 GB, if you do not increase it manually.

In one of our products we followed this solution in our deploy script:

 node --max-old-space-size=4096 yourFile.js 

There would also be a new space command but as I read here: a-tour-of-v8-garbage-collection the new space only collects the newly created short-term data and the old space contains all referenced data structures which should be in your case the best option.

回答2:

I encountered this issue when trying to debug with VSCode, so just wanted to add this is how you can add the argument to your debug setup.

You can add it to the runtimeArgs property of your config in launch.json.

See example below.

{ "version": "0.2.0", "configurations": [{         "type": "node",         "request": "launch",         "name": "Launch Program",         "program": "${workspaceRoot}\\server.js"     },     {         "type": "node",         "request": "launch",         "name": "Launch Training Script",         "program": "${workspaceRoot}\\training-script.js",         "runtimeArgs": [             "--max-old-space-size=4096"         ]     } ]} 

回答3:

fwiw, finding and fixing a memory hog with something like memwatch might help.

回答4:

i was struggling with this even after setting --max-old-space-size.

Then i realised need to put options --max-old-space-size before the karma script.

also best to specify both syntaxes --max-old-space-size and --max_old_space_size my script for karma :

node --max-old-space-size=8192 --optimize-for-size --max-executable-size=8192  --max_old_space_size=8192 --optimize_for_size --max_executable_size=8192 node_modules/karma/bin/karma start --single-run --max_new_space_size=8192   --prod --aot 

reference https://github.com/angular/angular-cli/issues/1652

回答5:

Upgrade node to the latest version. I was on node 6.6 with this error and upgraded to 8.9.4 and the problem went away.

回答6:

Try running this command:

npm rebuild grpc 

文章来源: Node.js heap out of memory