I have a main Python process, and a bunch or workers created by the main process using os.fork().
I need to pass large and fairly involved data structures from the workers back to the main process. What existing libraries would you recommend for that?
The data structures are a mix of lists, dictionaries, numpy arrays, custom classes (which I can tweak) and multi-layer combinations of the above.
Disk I/O should be avoided. If I could also avoid creating copies of the data -- for example by having some kind of shared-memory solution -- that would be nice too, but is not a hard constraint.
For the purposes of this question, it is mandatory that the workers are created using
os.fork(), or a wrapper thereof that would clone the master process's address space.
This only needs to work on Linux.
multiprocessing's queue implementation works.
q = multiprocessing.Queue() if (os.fork() == 0): print(q.get()) else: q.put(5) # outputs: 5