Gevent monkeypatching breaking multiprocessing

Question

I am attempting to use multiprocessing\'s pool to run a group of processes, each of which will run a gevent pool of greenlets. The reason for this is that there is a lot of

Answer 1

Wrote a replacement Nose Multiprocess plugin - this one should play well with all kinds of crazy Gevent-based patching.

https://pypi.python.org/pypi/nose-gevented-multiprocess/

https://github.com/dvdotsenko/nose_gevent_multiprocess

• Switches from multiprocess.fork to plain subprocess.popen for worker processes (fixes module-level erroneously shared objects issues for me)
• Switched from multiprocess.Queue to JSON-RPC over HTTP for master-to-clients RPC
• This can now theoretically allow tests to be distributed to multiple machines

Answer 2

use monkey.patch_all(thread=False, socket=False)

I have run into the same issue in a similar situation and tracked this down to line 115 in gevent/monkey.py under the patch_socket() function: _socket.socket = socket.socket. Commenting this line out prevents the breakage.

This is where gevent replaces the stdlib socket library with its own. multiprocessing.connection uses the socket library quite extensively, and is apparently not tolerant to this change.

Specifically, you will see this in any scenario where a module you import performs a gevent.monkey.patch_all() call without setting socket=False. In my case it was grequests that did this, and I had to override the patching of the socket module to fix this error.

Answer 3

Your provided code works for me on Windows 7.

EDIT:

Removed previous answer, because I've tried your code on Ubuntu 11.10 VPS, and I'm getting the same error.

Look's like Eventlet have this issue too

Answer 4

If you will use original Queue, then you code will work normally even with monkey patched socket.

import multiprocessing

from gevent import monkey
monkey.patch_all(thread=False)

q= multiprocessing.Queue()

Answer 5

Application of multiprocessing in the context of gevent is unfortunately known to raise problems. Your rationale, however, is reasonable ("a lot of network activity, but also a lot of CPU activity"). If you like, have a look at http://gehrcke.de/gipc. This is designed primarily for your use case. With gipc, you can easily spawn a few fully gevent-aware child processes and let them cooperatively talk to each other and/or with the parent through pipes.

If you have specific questions, you're welcome to get back to me.