How do I pass large numpy arrays between python subprocesses without saving to disk?

Question

Is there a good way to pass a large chunk of data between two python subprocesses without using the disk? Here\'s a cartoon example of what I\'m hoping to accomplish:

<

Answer 1

Use threads. But I guess you are going to get problems with the GIL.

Instead: Choose your poison.

I know from the MPI implementations I work with, that they use shared memory for on-node-communications. You will have to code your own synchronization in that case.

2 GB/s sounds like you will get problems with most "easy" methods, depending on your real-time constraints and available main memory.