Shutdown socketserver serve_forever() in one-thread Python application

I know that socketserver has a method shutdown() which causes server to shut down but this only works in multiple threads application since the shutdown needs to be called from different thread than the thread where serve_forever() is running.

My application handles only one request at time so I don't use separate threads for handling requests and I am unable to call shutdown() because it causes deadlock (it's not in the docs but it's stated directly in the source code of socketserver).

I'll paste here simplified version of my code for better understanding:

import socketserver

class TCPServerV4(socketserver.TCPServer):
  address_family = socket.AF_INET
  allow_reuse_address = True

class TCPHandler(socketserver.BaseRequestHandler):
  def handle(self):
    try:
       data = self.request.recv(4096)
    except KeyboardInterrupt:
       server.shutdown()

server = TCPServerV4((host, port), TCPHandler)
server.server_forever()

I am aware that this code is not working. I just wanted to show you the thing I'd like to accomplish - to shutdown server and quit the application while waiting for incoming data when the user presses Ctrl-C.

dmitry_romanov

You can start another thread locally, in your handler, and call shutdown from there.

Working demo:

    #!/usr/bin/env python
    # -*- coding: UTF-8 -*-

    import SimpleHTTPServer
    import SocketServer
    import time
    import thread

    class MyHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
        def do_POST(self):
            if self.path.startswith('/kill_server'):
                print "Server is going down, run it again manually!"
                def kill_me_please(server):
                    server.shutdown()
                thread.start_new_thread(kill_me_please, (httpd,))
                self.send_error(500)

    class MyTCPServer(SocketServer.TCPServer):
        def server_bind(self):
            import socket
            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            self.socket.bind(self.server_address)

    server_address = ('', 8000)
    httpd = MyTCPServer(server_address, MyHandler)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()

Few notes:

To kill server, do POST request to http://localhost:8000/kill_server.
I create function which calls server.shutdown() and run it from another thread to solve the problem we discuss.
I use advice from https://stackoverflow.com/a/18858817/774971 to make socket instantly avaliable for reuse (you can run server again without having [Errno 98] Address already in use error). With stock TCPServer you will have to wait for connection to timeout to run you server again.

The SocketServer library uses some weird ways of handling inherited attributes ( guessing due to use of old style classes). If you create a server and list it's protected attributes you see:

In [4]: server = SocketServer.TCPServer(('127.0.0.1',8000),Handler)
In [5]: server._

server._BaseServer__is_shut_down
server.__init__
server._BaseServer__shutdown_request
server.__module__
server.__doc__
server._handle_request_nonblock

If you just add the following in your request handler:

self.server._BaseServer__shutdown_request = True

The server will shutdown. This does the same thing as calling server.shutdown(), just without waiting (and deadlocking the main thread) until it's shutdown.

I believe the OP's intent is to shut down the server from the request handler and I think that the KeyboardInterrupt aspect of his code is just confusing things.

Pressing ctrl-c from the shell where the server is running will succeed in shutting it down without doing anything special. You can't press ctrl-c from a different shell and expect it to work, and I think that notion may be where this confusing code is coming from. There is no need to handle the KeyboardInterrupt in handle() as the OP tried, or around serve_forever() as another suggested. If you do neither, it works as expected.

The only trick here -- and it is tricky -- is telling the server to shutdown from the handler without deadlocking.

As the OP explained and showed in his code, he is using a single threaded server, so the user who suggested to shut it down in the "other thread" is not paying attention.

I dug around the SocketServer code and discovered that the BaseServer class, in its effort to work with the threaded mixins available in this module, actually makes it more difficult to use with non-threaded servers, by using a threading.Event around the loop in serve_forever.

So, I wrote a modified version of serve_forever for single-threaded servers which makes it possible to shut down the server from the request handler.

import SocketServer
import socket
import select

class TCPServerV4(SocketServer.TCPServer):
    address_family = socket.AF_INET
    allow_reuse_address = True

    def __init__(self, server_address, RequestHandlerClass):
        SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass)
        self._shutdown_request = False

    def serve_forever(self, poll_interval=0.5):
        """provide an override that can be shutdown from a request handler.
        The threading code in the BaseSocketServer class prevented this from working
        even for a non-threaded blocking server.
        """
        try:
            while not self._shutdown_request:
                # XXX: Consider using another file descriptor or
                # connecting to the socket to wake this up instead of
                # polling. Polling reduces our responsiveness to a
                # shutdown request and wastes cpu at all other times.
                r, w, e = SocketServer._eintr_retry(select.select, [self], [], [],
                                       poll_interval)
                if self in r:
                    self._handle_request_noblock()
        finally:
            self._shutdown_request = False

class TCPHandler(SocketServer.BaseRequestHandler):
    def handle(self):
        data = self.request.recv(4096)
        if data == "shutdown":
            self.server._shutdown_request = True

host = 'localhost'
port = 52000
server = TCPServerV4((host, port), TCPHandler)
server.serve_forever()

If you send the string 'shutdown' to the server, the server will end its serve_forever loop. You can use netcat to test this:

printf "shutdown" | nc localhost 52000

You should call shutdown in other thread actually as pointed in source code:

 def shutdown(self):
        """Stops the serve_forever loop.

        Blocks until the loop has finished. This must be called while
        serve_forever() is running in another thread, or it will
        deadlock.
        """
        self.__shutdown_request = True
        self.__is_shut_down.wait()

Ondergetekende

If you don't catch the KeyboardInterrupt in the tcp handler (or if you re-raise it), it should trickle down to the root call, in this case the server_forever() call.

I haven't tested this, though. The code would look like this:

import socketserver  # Python2: SocketServer

class TCPServerV4(socketserver.TCPServer):
    address_family = socket.AF_INET
    allow_reuse_address = True

class TCPHandler(socketserver.BaseRequestHandler):
    def handle(self):
        data = self.request.recv(4096)

server = TCPServerV4((host, port), TCPHandler)
try:
    server.serve_forever()
except KeyboardInterrupt:
    server.shutdown()

You could always try signals:

import signal import os

# inside handler code that has decided to shutdown:

os.kill(os.getpid(),signal.SIGHUP) # send myself sighup

来源：https://stackoverflow.com/questions/10085996/shutdown-socketserver-serve-forever-in-one-thread-python-application

标签

python

sockets

socketserver