Pytorch multiprocessing with shared memory causes matmul to be 30x slower (with only two processes)

Question

I am trying to improve the speed of my reinforcement learning algorithm by using multiprocessing to have multiple workers generating experience at the same time. Each proce