Node JS app crashing itself even after using forever JS

Question

I am just tired of seeing my server goes down every time off and on.. Whenever I wake up it is down. Every time I have to go to SSH and then run

forever restartall 

to restart my node app :(

I looked into the log file of forever and it says this

error: Forever detected script was killed by signal: SIGKILL
error: Script restart attempt #1

I have absolutely no idea why is this happening.

I am even running a cron job after every 50 minutes to auto restart node js in case it went down itself

0,50 * * * * * forever restartall

My app is running on Ubuntu EC2 AWS. And I use the following command to start my node app using forever:

forever -m5000 -w start index.js

Also, here are some recent snap of syslog since this happened recently

Jun  1 12:50:01 ip-172-31-28-35 CRON[25924]: (ubuntu) CMD (forever restartall)
Jun  1 12:50:01 ip-172-31-28-35 CRON[25923]: (CRON) info (No MTA installed, discarding output)
Jun  1 13:00:01 ip-172-31-28-35 CRON[25930]: (ubuntu) CMD (forever restartall)
Jun  1 13:00:01 ip-172-31-28-35 CRON[25929]: (CRON) info (No MTA installed, discarding output)
Jun  1 13:05:50 ip-172-31-28-35 dhclient: DHCPREQUEST of 172.31.28.35 on eth0 to 172.31.16.1 port 67 (xid=0x58e67545)
Jun  1 13:05:50 ip-172-31-28-35 dhclient: DHCPACK of 172.31.28.35 from 172.31.16.1
Jun  1 13:05:50 ip-172-31-28-35 dhclient: bound to 172.31.28.35 -- renewal in 1415 seconds.
Jun  1 13:09:01 ip-172-31-28-35 CRON[26000]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Jun  1 13:17:01 ip-172-31-28-35 CRON[26016]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jun  1 13:20:01 ip-172-31-28-35 CRON[26023]: (ubuntu) CMD (/home/ubuntu/cron/serv.sh)
Jun  1 13:20:01 ip-172-31-28-35 CRON[26022]: (CRON) info (No MTA installed, discarding output)
Jun  1 13:29:25 ip-172-31-28-35 dhclient: DHCPREQUEST of 172.31.28.35 on eth0 to 172.31.16.1 port 67 (xid=0x58e67545)
Jun  1 13:29:25 ip-172-31-28-35 dhclient: DHCPACK of 172.31.28.35 from 172.31.16.1
Jun  1 13:29:25 ip-172-31-28-35 dhclient: bound to 172.31.28.35 -- renewal in 1560 seconds.
Jun  1 13:34:39 ip-172-31-28-35 crontab[26289]: (ubuntu) LIST (ubuntu)

Here is the snapshot of the free-h command on my terminal of Ubuntu:

and here is snapshot of df -h command:

Is there any way to fix this and diagnose why my app crashes itself?

LATEST EDIT AND LOGS AFTER FOLLOWING SUGGESTIONS: Uninstalled MySQL. Added SWAP. I added the uncaught exception code Now when I woke up today the server was again down and this is my forever log http://kl1p.com/AI01 and these r my free-h just after the crash and syslog screenshot https://snag.gy/WMzqL0.jpg https://snag.gy/0wG8Dx.jpg

Can anyone please help what is causing the RAM to go fully used and why Node JS is causing errors, how to fix them?

Answer 1

I suspect memory issues. Brute forcing forever restarts (cron's etc) will only mask the real issue.

Add this to your code:

process.on('uncaughtException', function (err) {
  console.log("Uncaught Exception:", err);
  process.exit(1);  // This is VITAL. Don't swallow the err and try to continue.
});

This will allow you to begin the process of diagnosing what is causing your node server to fail.