What is the subprocess.Popen max length of the args parameter?

Question

I am using Popen function from the subprocess module to execute a command line tool:

subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=No         


        

Answer 1

For Unix-like platforms, the kernel constant ARG_MAX is defined by POSIX. It is required to be at least 4096 bytes, though on modern systems, it's probably a megabyte or more.

On many systems, getconf ARG_MAX will reveal its value at the shell prompt.

The shell utility xargs conveniently allows you to break up a long command line. For example, if

python myscript.py *

fails in a large directory because the list of files expands to a value whose length in bytes exceeds ARG_MAX, you can work around it with something like

printf '%s\0' * |
xargs -0 python myscript.py

(The option -0 is a GNU extension, but really the only completely safe way to unambiguously pass a list of file names which could contain newlines, quoting characters, etc.) Maybe also explore

find . -maxdepth 1 -type f -exec python myscript.py {} +

Conversely, to pass a long list of arguments to subprocess.Popen() and friends, something like

p = subprocess.Popen(['xargs', '-0', 'command'],
    stdin=subprocess.PIPE, stdout=subprocess.PIPE,
    stderr=subprocess.PIPE)
out, err = p.communicate('\0'.join(long_long_argument_list))

... where in most scenarios you should probably avoid raw Popen() and let a wrapper function like run() or check_call() do most of the work:

r = subprocess.run(['xargs', '-0', 'command'],
    input='\0'.join(long_long_argument_list),
    universal_newlines=True)
out = r.stdout

subprocess.run() supports text=True in 3.7+ as the new name of universal_newlines=True. Older Python versions than 3.5 didn't have run, so you need to fall back to the older legacy functions check_output, check_call, or (rarely) call.