Python fork(): passing data from child to parent

走远了吗. 提交于 2019-12-04 19:23:22


I have a main Python process, and a bunch or workers created by the main process using os.fork().

I need to pass large and fairly involved data structures from the workers back to the main process. What existing libraries would you recommend for that?

The data structures are a mix of lists, dictionaries, numpy arrays, custom classes (which I can tweak) and multi-layer combinations of the above.

Disk I/O should be avoided. If I could also avoid creating copies of the data -- for example by having some kind of shared-memory solution -- that would be nice too, but is not a hard constraint.

For the purposes of this question, it is mandatory that the workers are created using os.fork(), or a wrapper thereof that would clone the master process's address space.

This only needs to work on Linux.


multiprocessing's queue implementation works.

q = multiprocessing.Queue()
if (os.fork() == 0):
# outputs: 5