Grails: Tomcat won't shut down cleanly in prod

Question

I have an issue with shutting down a Grails app in production.

It shuts down cleanly when running from IntelliJ. But on a standalone Tomcat 7, shutting down gets it into a zombie state where the java process still exists but HTTP requests hang. I have to kill the java process (using kill).

I'm using Tomcat's standard bin/startup.sh and shutdown.sh. With Tomcat stopped, I drop the .war into Tomcat's /webapps directory and then start.

I suspect it could be the Quartz job scheduler plugin, but I deployed a version with no jobs in grails-app/jobs and it still hangs.

Anybody run across this before? Thanks!

Answer 1

Non-Daemon Threads

Probably one or more non-daemon thread is still running and preventing tomcat from a successful shutdown.

• open Terminal and type ps -ef| grep java and find your Tomcat7 p_id
• type kill -3 p_id
• type (in your Tomcat directory) tail -200 logs/catalina.out
• inspect thread dump generated by the kill -3
• look for non-daemon threads
• inspect your code to determine why that process is still alive.

Answer 2

I agree with Kerem Baydoğan. There's likely a non-daemon thread which doesn't want to stop. A thread dump is a great way to track this down.

I would recommend connecting to your remote JVM using VisualVM. This has been an invaluable tool for me to gather information about my running applications. If you're on a Mac, it's installed as part of the OS already. From a terminal, just type 'jvisualvm'. If not on a Mac, it's a free download from http://visualvm.java.net/

Once installed, you may need to add a few JVM args to allow the remote connection. http://visualvm.java.net/jmx_connections.html

Screenshot: