Unable to Run Multiple Node Child Processes without Choking on DigitalOcean

Question

I\'ve been struggling to run multiple instances of Puppeteer on DigitalOcean for quite some time with little luck. I\'m able to run ~5 concurrently using tools like puppetee

Answer 1

Most probably you're running out of memory, 5 puppeteer processes are a lot for a 1GB VM.

You can run

grep -i 'killed process' /var/log/messages

to confirm that the OOM killer terminated your processes.

Answer 2

Author of puppeteer-cluster here. You are right, 1 GB of memory is likely not enough for running 5 browser windows (or tabs) in addition to your operating system and maybe even other background tasks.

Here is a list of resources you should check:

• Memory: Use a tool like htop to check your memory usage while your application is running.
• CPU: Again, you can use htop for that, 3 vCPUs should be more than enough for 5 windows.
• Disk space: Use a tool like df to check if there is enough space on the disk. I know of multiple cases in which there was not enough space on the disk (like some old kernels filling the disk), and Chrome needs at least some space to run.
• Network throughput: Rarely the problem, but sometimes the network just does not have the bandwidth to support many open browser. Use a tool like nload to check the network throughput.

To use htop or nload, you start your script in the background (node script.js &) or use a terminal multiplexer (like tmux). Resource problems should then be easy to spot.