Python multiprocessing: RuntimeError: “Queue objects should only be shared between processes through inheritance”

你说的曾经没有我的故事 提交于 2019-12-04 03:51:29

问题


I am aware of multiprocessing.Manager() and how it can be used to create shared objects. In particular, queues which can be shared among workers. There is this question, this question, and this question.

However, these links don't mention why we can use inheritance for sharing between processes. As I understand, a queue can still only be copied in this case.


回答1:


The Queue implementation in python relies on a system pipe to transmit the data from one process to another and some semaphores to protect the read and write on this pipe.

The pipe is handled as an open file in the process and can only be shared with a subprocess at spawning time, because of OS constraints.
The semaphores are also treated as files that should only be shared at spawning time, at least in UNIX based system, for early version of python.

As these 2 sub objects cannot be shared in general, the Queue cannot be pickled and sent to a subprocess once it has been started.

However, for some OS and recent version of python, it is possible to share the Connection and to create sharable Semaphore. Thus, you could in theory create your own Queue that can be shared between processes. But it involves a lot of hacks and might not be very secured.



来源:https://stackoverflow.com/questions/42638147/python-multiprocessing-runtimeerror-queue-objects-should-only-be-shared-betwe

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!