Automatically restarting HHVM when it stops responding but process not dead

Question

I'm having a problem where every 12-24 hours, HHVM crashes but it seems to leave the process running. It seems most providers just use php5-fpm as a failover within nginx for stability. However, this won't restart the non-responsive hhvm instance.

Since the process is left running, most server monitoring solutions will see it as a live daemon, and not restart it. HTTP monitoring can be slow to react.

Is it possible to trigger the hhvm restart on failover? If not, what would be the best solution to ensure a listening daemon that's non responsive is restarted.

Answer 1

kristapsk posted a solution on Official bug regarding this topic. This is the script he told to add to cron to be executed every 2 minutes.

I have modified it a bit to make it work with prebuilt packages of HHVM.

#! /bin/bash
PID="`cat /var/run/hhvm/pid`"

if [ "$PID" == "" ]; then
    echo No PID, starting up
    /etc/init.d/hhvm start
else
    if [ "`ps ax -o pid | grep $PID`" == "" ]; then
        echo HHVM PID $PID not running, starting up
        # Stop, just in case, if crashed. Else you would get:
        #  * WARNING: hhvm.83 has already been started
        /etc/init.d/hhvm stop
        /etc/init.d/hhvm start
    fi
fi

Answer 2

Regarding your comment on the other answer, a good option when you have a fallback to php and you want to ensure that HHVM is the one that responded is to use the solution from the answer here with one change, in your status script do a check for:

if (defined('HHVM_VERSION')) {
  echo 'status';
}

The two solutions combined will handle all of hhvm's possible ways of crashing.