Performance of: asynchronous request handler with blocking tasks handled by worker pool

Question

How is the performance of this script: http://tornadogists.org/2185380/ copied below.

from time import sleep
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.web import Application, asynchronous, RequestHandler
from multiprocessing.pool import ThreadPool

_workers = ThreadPool(10)

def run_background(func, callback, args=(), kwds={}):
    def _callback(result):
        IOLoop.instance().add_callback(lambda: callback(result))
    _workers.apply_async(func, args, kwds, _callback)

# blocking task like querying to MySQL
def blocking_task(n):
    sleep(n)
    return n

class Handler(RequestHandler):
    @asynchronous
    def get(self):
        run_background(blocking_task, self.on_complete, (10,))

    def on_complete(self, res):
        self.write("Test {0}<br/>".format(res))
        self.finish()

HTTPServer(Application([("/", Handler)],debug=True)).listen(8888)
IOLoop.instance().start()

1. My application will have way over 1,000 req/sec.
2. Each request will last from 2-30 seconds, averaging about 6 seconds
   • Simply averaging sleep(6)
3. Block IO by using something like redis BLPOP or Queue.get_nowait()

Answer 1

The overall pattern is fine, with the caveat that thanks to the GIL, your thread pool will only be able to use a single CPU, and you'll need to use multiple processes to make full use of the available hardware.

Taking a closer look at the numbers, 10 threads is way too small if your requests are really going to average 6 seconds each. You've got 6000 seconds worth of work coming in every second, so you need a total of at least 6000 threads across all your processes (and that's assuming the 6 seconds is really just blocking on external events and the CPU cost in the python process is negligible). I'm not sure how many threads a modern system can handle, but 6000 Python threads doesn't sound like a great idea. If you've really got 6 seconds of blocking per request (and thousands of requests/sec) it sounds like it would be worthwhile to convert these blocking functions to be asynchronous.