How can I profile a multithread program in Python?

Question

I\'m developing an inherently multithreaded module in Python, and I\'d like to find out where it\'s spending its time. cProfile only seems to profile the main thread. Is the

Answer 1

Instead of running one cProfile, you could run separate cProfile instance in each thread, then combine the stats. Stats.add() does this automatically.

Answer 2

Given that your different threads' main functions differ, you can use the very helpful profile_func() decorator from here.

Answer 3

I don't know any profiling-application that supports such thing for python - but You could write a Trace-class that writes log-files where you put in the information of when an operation is started and when it ended and how much time it consumed.

It's a simple and quick solution for your problem.

Answer 4

Check out mtprof from the Dask project:

https://github.com/dask/mtprof

It's a drop-in replacement for cProfile that, if your threads are launched in the usual way and complete before your main thread, will roll-up their stats into the same reporting stats. Worked like a charm for me.

Answer 5

If you're okay with doing a bit of extra work, you can write your own profiling class that implements profile(self, frame, event, arg). That gets called whenever a function is called, and you can fairly easily set up a structure to gather statistics from that.

You can then use threading.setprofile to register that function on every thread. When the function is called you can use threading.currentThread() to see which it's running on. More information (and ready-to-run recipe) here:

http://code.activestate.com/recipes/465831/

http://docs.python.org/library/threading.html#threading.setprofile

Answer 6

Please see yappi (Yet Another Python Profiler).