Python: ulimit and nice for subprocess.call / subprocess.Popen?

Question

I need to limit the amount of time and cpu taken by external command line apps I spawn from a python process using subprocess.call , mainly because sometimes the spawned pro

Answer 1

Erik made it easy for me, but he forgot the nice part which Rich Pointed out. I find the psutil package nice(pun intended) but unfortunately less portable. Here is my take at the question:

import os
import psutil
import resource
import subprocess

def preexec_fn():
    pid = os.getpid()
    ps = psutil.Process(pid)
    ps.set_nice(10)
    resource.setrlimit(resource.RLIMIT_CPU, (1, 1))

print "mother pid", os.getpid()
p = subprocess.Popen(["./cpuhog.sh"], preexec_fn=preexec_fn)
p.wait()
print "mother still alive with pid", os.getpid()

Ville used the shell=True to which I'm somehow allergic. Perhaps I'm just old and grumpy here, but I try to avoid it!

Answer 2

You can set limits for subprocesses with the ulimit and nice shell commands like this:

import subprocess
subprocess.Popen('ulimit -t 60; nice -n 15 cpuhog', shell=True)

This runs cpuhog with a limit of 60 seconds of CPU time and a niceness adjustment of 15. Note that there is no simple way to set a 20% CPU throttle as such. The process will use 100% CPU unless another (less nice) process also needs the CPU.

Answer 3

Use the preexec_fn parameter to subprocess.Popen, and the resource module. Example:

parent.py:

#!/usr/bin/env python

import os
import sys
import resource
import subprocess

def setlimits():
    # Set maximum CPU time to 1 second in child process, after fork() but before exec()
    print "Setting resource limit in child (pid %d)" % os.getpid()
    resource.setrlimit(resource.RLIMIT_CPU, (1, 1))

print "CPU limit of parent (pid %d)" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)
p = subprocess.Popen(["./child.py"], preexec_fn=setlimits)
print "CPU limit of parent (pid %d) after startup of child" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)
p.wait()
print "CPU limit of parent (pid %d) after child finished executing" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)

child.py:

#!/usr/bin/env python

import os
import sys
import resource

print "CPU limit of child (pid %d)" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)

parent.py will fork into a new process. In the new process, it will call setlimits(), then exec child.py. This means the resource will be limited in the child process, but not in the parent.

Output when running program:

./parent.py
CPU limit of parent (pid 17404) (-1, -1)
Setting resource limit in child (pid 17405)
CPU limit of parent (pid 17404) after startup of child (-1, -1)
CPU limit of child (pid 17405) (1, 1)
CPU limit of parent (pid 17404) after child finished executing (-1, -1)

This is in many cases a better solution than trying to use ulimit, since it's not always a good idea to spawn subprocess via shell, especially since it often causes ugly parameter quoting trouble.