Jetty: Stopping programatically causes “1 threads could not be stopped”

Question

I have an embedded Jetty 6.1.26 instance. I want to shut it down by HTTP GET sent to /shutdown. So I created a JettyShutdownServlet:

@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {

 resp.setStatus(202, "Shutting down.");
 resp.setContentType("text/plain");
 ServletOutputStream os = resp.getOutputStream();
 os.println("Shutting down.");
 os.close();
 resp.flushBuffer();

 // Stop the server.
 try {
    log.info("Shutting down the server...");
    server.stop();
 } catch (Exception ex) {
    log.error("Error when stopping Jetty server: "+ex.getMessage(), ex);
 }

However, when I send the request, Jetty does not stop - a thread keeps hanging in org.mortbay.thread.QueuedThreadPool on the line with this.wait():

   // We are idle
   // wait for a dispatched job
   synchronized (this)
   {
       if (_job==null)
          this.wait(getMaxIdleTimeMs());
       job=_job;
       _job=null;
   }

...

2011-01-10 20:14:20,375 INFO  org.mortbay.log jetty-6.1.26
2011-01-10 20:14:34,756 INFO  org.mortbay.log Started SocketConnector@0.0.0.0:17283
2011-01-10 20:25:40,006 INFO  org.jboss.qa.mavenhoe.MavenHoeApp Shutting down the server...
2011-01-10 20:25:40,006 INFO  org.mortbay.log Graceful shutdown SocketConnector@0.0.0.0:17283
2011-01-10 20:25:40,006 INFO  org.mortbay.log Graceful shutdown org.mortbay.jetty.servlet.Context@1672bbb{/,null}
2011-01-10 20:25:40,006 INFO  org.mortbay.log Graceful shutdown org.mortbay.jetty.webapp.WebAppContext@18d30fb{/jsp,file:/home/ondra/work/Mavenhoe/trunk/target/classes/org/jboss/qa/mavenhoe/web/jsp}
2011-01-10 20:25:43,007 INFO  org.mortbay.log Stopped SocketConnector@0.0.0.0:17283
2011-01-10 20:25:43,009 WARN  org.mortbay.log 1 threads could not be stopped
2011-01-10 20:26:43,010 INFO  org.mortbay.log Shutdown hook executing
2011-01-10 20:26:43,011 INFO  org.mortbay.log Shutdown hook complete

It blocks for exactly one minute, then shuts down. I've added the Graceful shutdown, which should allow me to shut the server down from a servlet; However, it does not work as you can see from the log.

I've solved it this way:

Server server = new Server( PORT );
server.setGracefulShutdown( 3000 );
server.setStopAtShutdown(true);
...
server.start();

if( server.getThreadPool() instanceof QueuedThreadPool ){
   ((QueuedThreadPool) server.getThreadPool()).setMaxIdleTimeMs( 2000 );
}

setMaxIdleTimeMs() needs to be called after the start(), becase the threadPool is created in start(). However, the threads are already created and waiting, so it only applies after all threads are used at least once.

I don't know what else to do except some awfulness like interrupting all threads or System.exit().

Any ideas? Is there a good way?

Answer 1

Graceful doesn't do what you think it does - it allows the server to shutdown gracefully, but it does not allow you to shutdown from inside a servlet.

The problem is as described in the mailing-list post you linked to - you're trying to stop the server, while you're still processing a connection inside the server.

You should try changing your servlet's implementation to:

// Stop the server.
new Thread()
{
   public void run() {
     try {
        log.info("Shutting down the server...");
        server.stop();
        log.info("Server has stopped.");
     } catch (Exception ex) {
        log.error("Error when stopping Jetty server: "+ex.getMessage(), ex);
     }
   }
}.start();

That way the servlet can finished processing while the server is shutting down, and will not hold up the shutdown process.