Python multiprocessing performance

Question

This should be my third and final question regarding my attempts to increase performance on some statistical analysis that I am doing with python. I have 2 versions of my code (

Answer 1

Shared data is a known case of slowdowns due to synchronization.

Can you split your data among processes, or give each process an independent copy? Then your processes would not need to synchronize anything up until the moment when all calculations are done.

Then I'd let the master process join the output of all worker processors into one coherent set.

The approach may take extra RAM, but RAM is cheap nowadays.

If you ask, I'm also puzzled by 3700 ms per thread lock acquisition. OTOH profiling may be mistaken about special calls like this.