Controlling scheduling priority of python threads?

Question

I\'ve written a script that uses two thread pools of ten threads each to pull in data from an API. The thread pool implements this code on ActiveState. Each thread pool is

Answer 1

It doesn't work, but I tried:

1. getting the parent pid and priority
2. launching threads using concurrent.futures.ThreadPoolExecutor
3. using ctypes to get the (linux) thread id from within the thread(works)
4. using the tid with os.setpriority(os.PRIO_PROCESS,tid,parent_priority+1)
5. calling pool.shutdown() from the parent.

Even with liberal sprinkling of os.sched_yield(), the child threads never actually run past the setpriority().

Reading man pages, it seems threads don't have the capability to change (even their) scheduling priority; you have to do something with "capabilities" to give the thread the "CAP_SYS_NICE" capability. Running the process with root permissions didn't help either; child threads still don't run.

Answer 2

The python threading-docs mention explicitly that there is no support for setting thread-priorities:

The design of this module is loosely based on Java’s threading model. However, where Java makes locks and condition variables basic behavior of every object, they are separate objects in Python. Python’s Thread class supports a subset of the behavior of Java’s Thread class; currently, there are no priorities, no thread groups, and threads cannot be destroyed, stopped, suspended, resumed, or interrupted. The static methods of Java’s Thread class, when implemented, are mapped to module-level functions.

Answer 3

I know, a lot of time has passed, but I recently came across this question, and I thought it would be useful to add another option.

Have a look at threading2, which is a drop-in replacement and extension for the default threading module, with support – sort of – for priority and affinity.

Answer 4

I believe that threading priority is not controllable in python due to how they are implemented using a global interpreter lock (GIL). Having said that, even if you could give one thread more CPU processing priority, the python implementation that hands around the GIL would not be aware of this as it handed around the GIL. If you were able to increase niceness in a single thread in your pool (say it is doing a more important job) you would need to use your own implementation of locks to give the higher priority thread access to the GIL more often.

A google search returns this article which I believe is similar to what you are asking

Explains why it doesnt work http://www.velocityreviews.com/forums/t329441-threading-priority.html

Explains the workaround I was suggesting http://bytes.com/topic/python/answers/645966-setting-thread-priorities

Answer 5

I was wondering if this answer at another related question might be useful in this scenario? (link)

As you are already using Subprocess.POpen to launch your PHP script, it strikes me that you can use "preexec_fn" and either a predefined function, or a lambda function (as demonstrated in the above linked answer) to set the nice level of each launched PHP thread?