First I will admit that there are a few to many keywords in that title, but I am indeed really trying to capture the problem in the correct manner. The issue here is that I can not seem to be able to correctly create a sub process using the python multiprocessing module without it causing the webpage response to hang. I have tried a few recent versions of gunicorn and the problem persists. Interestingly, the problem never was an issue on ubuntu server, but now moving the application to rhel6.5 this issue has presented itself. Here is the workflow:
-route is hit -form is submitted which hits the route and triggers a multiprocessing.Process() to be created, where the work done is to sleep for 30 seconds -the route appears to finish, as a print statement after the multiprocessing call is printed, however the browser keeps the connection open and does not 'finish loading' (show the page) until the 30 seconds of sleep are finished
Note that the form submission is not part of this issue, it just helps in viewing the issue happen.
Here is a very simple route and function that produces the issue:
def multi(): print 'begin multi' time.sleep(30) print 'end multi' @app.route('/multiprocesstest', methods=['GET','POST']) def multiprocesstest(): syntaxForm = forms.mainSyntaxForm() if syntaxForm.validate_on_submit(): print 'before multi call' th = multiprocessing.Process(target=multi) th.start() print 'after multi call' return redirect('multiprocesstest') return render_template('syntax_main.html', form=syntaxForm)
After extended research and sparse google results for this problem, I have not found anything conclusive. I am going to try another load balancer to check is the problem is gunicorn only.
OK, so I ran into this on Flask/UWSGI/NGINX/Linux 14.04.
Good news for future readers: I was able to resolve this. Bad news: I am pretty sure this is a horrible hack.
Some interesting test code to prove that this will hang forever:
@app.route('/spin-up') def spin_up(): import multiprocessing as mp def spin_forever(): while True: time.sleep(1) print('starting new process') p = mp.Process(target=spin_forever) print('created') p.start() print('started--generating result') return flask.render_template_string('awesome')
If you hit the endpoint '/spin-up' it will spin up the process and hang forever. Pretty awesome, huh?
Basic message-queues don't work unless you go fully out-of-process (i.e., use a message queue running in an already-started different process) or don't validate the success (i.e., wait for a success ACK response).
The short answer is that if you attempt to validate that your sub-process succeeded, you are in trouble. I was using an internal message queue between threads, and if I waited for my "success" response, the Flask server would still hang. E.g.,
@app.route('/spin-up') def spin_up(): put_start_message_on_queue(): while True: if got_success_response_from_queue(): break time.sleep(0.1) return flask.render_template_string('awesome')
This still hangs (forever), so instead I added a second command to the message queue called 'restart':
@app.route('/spin-up') def spin_up(): put_start_message_on_queue() while True: if got_success_message_from_queue(): break time.sleep(0.1) put_restart_message_on_queue() return flask.render_template_string('awesome')
You have to ensure that the restart_message, when received kills the existing process, then does a minimal amount of work before starting the new one, potentially even inserting a
time.sleep(0)
in the message handler. THIS IS A HUGE HACK as far as I can tell, but it works consistently, and so long as the process can be restarted (sorry if that's not the case for you...)
My program also hung in a similar fashion. Gevent and Multiprocessing.Process do not play well together, it seems.
Replace multiprocessing by multiprocessing.dummy might solve the issue since both gunicorn and multiprocessing are multiprocessing module and it might cause trouble when you try to invoke multiple processes inside of a single process.
