Memory leak in express.js api application

Question

I am running an express.js application, which is used as a REST api. One endpoint starts puppeteer and test my website with several procedures.

After starting the application and the continuous consumption of the endpoint, my docker container runs out of memory every hour as you can see below.

First, I thought I have a memory leak in my puppeteer / headless chrome, but I then I monitored the memory usage from the processes, there isn't and memory leak visible as you can see here:

     0.00 Mb COMMAND 
   384.67 Mb /var/express/node_modules/puppeteer/.local
   157.41 Mb node /var/express/bin/www 
   101.76 Mb node /usr/local/bin/pm2
     4.34 Mb /var/express/node_modules/puppeteer/.local
     1.06 Mb ps 
     0.65 Mb bash 
     0.65 Mb bash 
     0.31 Mb cut 
     0.31 Mb cut 
     0.13 Mb dumb

Now, I ran out of ideas what the problem could be. Has anyone an idea where the RAM consumption could came from?

Answer 1

Analyse the problem more

You need to monitor the activity real time.

We do not have the code, thus we cannot even know what is going on. However, you can use more advanced tool like htop, gtop, netdata and others than top or ps.

The log on pm2 might also tell you more about things. On such situation, the logs will have more data than the process manager. Make sure to thoroughly investigate the logs to see if scripts are responsible, and throwing errors or not,

pm2 logs

Each api call will cost you

Calculate the cost early and prepare accordingly,

• If you have 1 call, then be prepared to have 100Mb-1GB or more each time. It will cost you just like a browser tab. The cost will be there as long as the tab is open.
• If the target website is heavy, then it will cost more. Some websites like Youtube will obviously cost you more.
• Any script running inside the browser tab will cost cpu and memory usage.
• Say each process is causing 300MB ram, If you don't close the process properly and start making API calls, then only 10 API call will be able to use 3GB ram pretty easily. It can add up pretty quickly.

Make sure to close the tabs

Whether the automation task is successful or not, make sure to properly use browser.close() to ensure the resource it is using gets free. Most of time we forget about such small things and it costs us.

Apply dumb init on docker to avoid ghost process

Something like dumb-init or tini can be used if you have a process that spawns new processes and you doesn't have good signal handlers implemented to catch child signals and stop your child if your process should be stopped etc.

Read more on this SO answer.

Answer 2

I've got the problem solved. It was caused by the underlying Kubernetes system, which wasn't configured with a resource limit on that specific container. Therefore, the container can consume as many memory as possible.

Now I've limited at 2GB an it looks like this: