Hazelcast prevents the JVM from terminating

Question

We use Hazelcast 2.6.2 in a legacy Java clustered application. When the application is stopped the JVM does not terminate any more. It seems that it is caused by Hazelcast threads not being flagged daemon. I did not find a way way through the Hazelcast API to flag them daemon.

Are there recommended solutions to prevent Hazelcast from preventing the JVM to terminate?

Regards

Answer 1

Looking at the Hazelcast Javadocs, I see that there is a shutdownAll(); method. To quote the javadocs:

Shuts down all running Hazelcast Instances on this JVM, including the default one if it is running. It doesn't shutdown all members of the cluster but just the ones running on this JVM.

If you aren't shutting it down I suspect there are non-daemon threads there that aren't being terminated that will keep the JVM from closing.

Answer 2

In Tomcat, I added a lifecycle listener via server.xml.

<Listener className="com.mycompany.StartupHandler" DEBUG="false"/>

My StartupHandler.class went in a jar in $TOMCAT_HOME/lib/, and it contains this snippet to trigger Hazelcast shutdown when Tomcat shutdown is detected:

public void lifecycleEvent(LifecycleEvent lifecycleEvent) {
    String eventType = lifecycleEvent.getType();
    if (eventType.equals(Lifecycle.BEFORE_STOP_EVENT)) {
        com.hazelcast.core.Hazelcast.shutdownAll();
    }
}

Other web servers should have similar shutdown hooks where you can invoke Hazelcast.shutdownAll() automatically.

See https://github.com/hazelcast/hazelcast/issues/718 for my original Hazelcast thread on the same topic. Thanks for the hint here to call Hazelcast.shutdownAll(). That idea combined with my Tomcat shutdown hook should be sufficient to resolve this problem for me, and hopefully you find it useful too.

Answer 3

Here is a Tomcat independent way to detect JVM shutdown. It uses a non-daemon thread to poll a daemon thread to detect when the JVM is shutting down (the JVM shuts down daemon threads automatically). Poll the static IS_SHUTDOWN flag in your code, or inline the call to Hazelcast.shutdownAll() here. Be careful of a race condition, because if you try to shutdown Hazelcast before it starts then it throws an exception in your log. The exception does not break anything, but it looks ugly.

/**
 * Non-Daemon thread monitors a sacrificial, low-priority, daemon thread
 * to detect when JVM is shutting down, so shutdown hooks can be invoked.
 * @author justin.cranford
 */
public class ThreadUtil {
    public static boolean IS_SHUTDOWN = false;  // threads can poll this flag, or inline below where flag is set
    private static final Thread DAEMON_THREAD = new Thread() {
        public void run() {
            while (true) {
                try {
                    Thread.sleep(Long.MAX_VALUE);   // sleep forever
                } catch(Exception e) {}
            }
        }
    };
    private static final Thread NONDAEMON_THREAD = new Thread() {
        public void run() {
            while (true) {
                if (!DAEMON_THREAD.isAlive()) { // poll forever, until daemon thread dies
                    ThreadUtil.IS_SHUTDOWN = true;
                    return;
                }
                try {
                    Thread.sleep(1000); // poll every 1000msec = 1sec
                } catch(Exception e) {}
            }
        }
    };
    static {
        DAEMON_THREAD.setName("ShutdownMonitorNonDaemonThread");
        DAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
        DAEMON_THREAD.setDaemon(true);
        DAEMON_THREAD.start();
        try {
            Thread.sleep(3000); // wait 3000msec = 3sec before monitoring
        } catch(Exception e) {}
        NONDAEMON_THREAD.setName("ShutdownMonitorDaemonThread");
        NONDAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
        NONDAEMON_THREAD.setDaemon(false);
        NONDAEMON_THREAD.start();
    }
}

Answer 4

As everyone said, hazelcastInstance.shutdownAll() is the solution.

But, I would to get a new feature in Hazelcast - provide a demon client as well. There are many use cases where cache instance needs to be shutdown as soon as application ends.