Why isn't fromfile-prefix-chars in Python argparse working?

Question

I\'m trying to use the fromfile-prefix-chars feature of argparse in Python to load all my command line arguments from a file, but it keeps complaining that I haven\'t specif

Answer 1

From the documentation:

Arguments read from a file must by default be one per line ... and are treated as if they were in the same place as the original file referencing argument on the command line. So in the example above, the expression ['-f', 'foo', '@args.txt'] is considered equivalent to the expression ['-f', 'foo', '-f', 'bar'].

In the example:

fp.write('-f\nbar')

So the file contains:

-f
bar

In other words, each of the file lines corresponds to one 'word' (blank separated) in the command line. --option1=foo is one word. --option1 foo is interpreted just as though it was quoted in the command line,eg. prog.py '--option1 foo' '--option2 1234'

The https://docs.python.org/dev/library/argparse.html#argparse.ArgumentParser.convert_arg_line_to_args has a custom function that will split lines on spaces. Experiment with that if you want to stick with the argument file.

---

import argparse

with open('args.txt', 'w') as fp:
    fp.write('--option1 foo\n--option2 1234')  # error
    # but works with modifed 'convert...'
    #fp.write('--option1=foo\n--option2=1234')  # works
    #fp.write('--option1\nfoo\n--option2\n1234') # works

def convert_arg_line_to_args(arg_line):
    for arg in arg_line.split():
        if not arg.strip():
            continue
        yield arg
"""
default line converter:
def convert_arg_line_to_args(self, arg_line):
    return [arg_line]
"""

def go():
   parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
   parser.convert_arg_line_to_args = convert_arg_line_to_args
   parser.add_argument("--option1")
   parser.add_argument("--option2", type=int, required=True)
   args = parser.parse_args(['@args.txt'])
   print args

if __name__ == "__main__":
    go()

---

@toes suggested using shlex to parse the file. shlex has a nice feature in that it strips of unnecessary quotes.

shlex can be used to split individual lines of the file.

def sh_split(arg_line):
    for arg in shlex.split(arg_line):
        yield arg
parser.convert_arg_line_to_args = sh_split

Or it can replace the whole @file read method (_read_args_from_files)- this should function the same as @toes answer, except that the @file string can be anywhere in the commandline (or even be repeated).

def at_read_fn(arg_strings):
    # expand arguments referencing files
    new_arg_strings = []
    for arg_string in arg_strings:
        if not arg_string or not arg_string.startswith('@'):
            new_arg_strings.append(arg_string)
        else:
            with open(arg_string[1:]) as args_file:
                arg_strings = shlex.split(args_file.read())
                new_arg_strings.extend(arg_strings)
    return new_arg_strings
parser._read_args_from_files = at_read_fn

Obviously a cleaner production version would modify these methods in an ArgumentParser subclass.

Answer 2

I think there's a better answer to this: use shlex.

if sys.argv[1].startswith('@'):
    args = parser.parse_args( shlex.split(open(sys.argv[1][1:]).read()) )
else:
    args = parser.parse_args()

This allows you to specify args in a file in a more natural way e.g., it allows using spaces or equals sign to specify your args on a single line as in:

arg1
arg2
--opt1 'foo'
--opt2='bar'

shlex.split splits this as you would expect:

['arg1', 'arg2', '--opt1', 'foo', '--opt2=bar']

The only thing this method doesn't have is that it expects the @file.txt to be the first argument.

Answer 3

Try to this way

# encoding: utf-8
import imp
import argparse


class LoadConfigAction(argparse._StoreAction):
    NIL = object()

    def __init__(self, option_strings, dest, **kwargs):
        super(self.__class__, self).__init__(option_strings, dest)
        self.help = "Load configuration from file"

    def __call__(self, parser, namespace, values, option_string=None):
        super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)

        config = imp.load_source('config', values)

        for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
            setattr(namespace, key, getattr(config, key))

Usage

parser.add_argument("-C", "--config", action=LoadConfigAction, default=None)
parser.add_argument("-H", "--host")

Example config (real is python file)

# Config example /etc/my.conf
import os

# Parameter definition
host = os.getenv("HOST", "127.0.0.1")

Answer 4

The problem is that, when specified in a file, each argument must have an '=' between it and the option name. While argparse is somewhat more flexible on that format when run from the command line (where space or = is ok), when run from the file it must have an '='.

So, a working argument file would be:

--option1=foo
--option2=1234

Something else to be aware of, be sure you don't have any extra whitespace at the end of the lines or that whitespace will get included with the option when argparse reads the file.