When to use threading and how many threads to use

Question

I have a project for work. We had written a module and there as a #TODO to implement threading to improve the module. I\'m a fairly new python programmer and decided to take

Answer 1

You're right to think that multithreading is a questionable move here for good reason. Multithreading, as it stands, is great and in the right applications can make a worlds difference in running times.

However, on the other hand, it also adds additional complexity to any program that implements it (especially in python). There are also time penalties to consider when using multithreading, such as those that occur when doing context switches or the time it takes to actually create a thread.

These time penalties are negligent when your program has to process thousands upon thousands of resource intense tasks because the time you would save from having multithreading far outweighs the little bit of time it takes to get the threads ready. For your case though, I'm not sure your needs meet those requirements. I didn't look to deep into what type of objects you were processing but you stated they only took about 2 seconds, which isn't awful and you also said that you only have 6 items at a time to process. So on average we can expect the main part of your scrip to run for 12 seconds. In my opinion, that is not necessary for multithreading because it will take a second or two to get the threads ready and then pass the instructions to them, whereas in one thread your python script would already be well into processing its second object in that time.

In short, I would save multithreading unless you need it. For example, huge datasets like those used for gene sequencing (big thing in Python) benefit greatly from it because multiple threads can help process these massive files concurrently. In your case, it doesn't look like the ends justify the means. Hope this helps