Using SimpleHTTPServer for unit testing

Question

I'm writing a Python module that wraps out a certain web service API. It's all REST, so relatively straightforward to implement.

However, I found a problem when it comes to unit testing: as I don't run the services I made this module for, I don't want to hammer them, but at the same time, I need to retrieve data to run my tests. I looked at SimpleHTTPServer, which would be OK.

I solved part of the issue I had, but now, since I can't seem to terminate the thread, I get issues with "address already in use" when launching the test application more than once.

Here's some sample code

PORT = 8001

handler = SimpleHTTPServer.SimpleHTTPRequestHandler

httpd = SocketServer.TCPServer(("", PORT), handler)

httpd_thread = threading.Thread(target=httpd.serve_forever)
httpd_thread.setDaemon(True)
httpd_thread.start()

api_data = urllib.urlopen("http://localhost:8001/post/index.json")
print "Data start:"
print json.load(api_data)

Where "index.json" is a mock JSON file I made which substitutes the real thing. How can I clean things gracefully after the program terminates?

Answer 1

Try using a subclass of TCPServer with allow_reuse_address set True:

class TestServer(SocketServer.TCPServer):
    allow_reuse_address = True

...
httpd = TestServer(("", PORT), handler)

Answer 2

We use a server built on wsgiref. http://docs.python.org/library/wsgiref.html

It's very easy to add features to this server as we add unit tests.

We start the server with subprocess. http://docs.python.org/library/subprocess.html?highlight=subprocess#module-subprocess

We do not use threads for this kind of testing. Why? (1) Our unit test server is rather complex and we'd like to keep it completely isolated from client applications. (2) Our client applications will be separate processes on separate hardware, we need to be sure that we have realistic performance expectations for that configuration. (3) It's simpler. (4) It's portable across all platforms. (5) It's trivial to change from stand-alone unit testing to integration testing with a production-like server that's already running.

We actually have a small WSGI application that makes the server shutdown in a reasonably controlled manner so that the logs are shutdown properly.

Answer 3

Old thread but the answers here didn't help me, I'm using HTTPServer, and shutting down after each unit test (by default HTTPServer has allow_reuse_address = 1 set). However I still got the address already in use problem after calling shutdown. I fixed using:

from BaseHTTPServer import HTTPServer

class MyHTTPServer(HTTPServer):
    def shutdown(self):
        self.socket.close()
        HTTPServer.shutdown(self)

Not sure why this doesn't happen by default? May be this isn't optimal?