How to use multicore python with PyQt4 process?

Question

I am writing an app in pyqt4 that has to read in and parse a lot of xml files. Done single-threaded it takes a while to do all that parsing and make the thousands of python objects corresponding to that incoming xml. I have profiled the code and as far as I can tell it's compute, not I/O, bound.

I would like to convert the app to a multi-core model to spread the load around, using a worker-farm model (?Process.Pool in python).

However, I would also like to be able to Signal progress from the workers to update the gui.

It seems to me from what I have read so far that QThread is not multicore capable (because it round-robins on one core) but I need QThread to do Signal, and so essentially I can't do that.

I might be able to arrange not to need to Signal from a worker, only from the farmer, which might then mean I can carry on, but then I wonder: can I return a list of python objects from one Process to another?

Answer 1

• Spawn a QThread.
• The QThread can farm out tasks to the multiprocessing Pool. You might use pool.apply_async() which has a callback parameter.
• The callback parameter allows you to specify a function which is called when the target function completes.
• The callback runs in the QThread, and is sent the return value of the target function as its one and only argument.
• Each time the callback function runs, you can update the GUI to indicate the progress.