Why isn't fromfile-prefix-chars in Python argparse working?

Question

I\'m trying to use the fromfile-prefix-chars feature of argparse in Python to load all my command line arguments from a file, but it keeps complaining that I haven\'t specif

Answer 1

From the documentation:

Arguments read from a file must by default be one per line ... and are treated as if they were in the same place as the original file referencing argument on the command line. So in the example above, the expression ['-f', 'foo', '@args.txt'] is considered equivalent to the expression ['-f', 'foo', '-f', 'bar'].

In the example:

fp.write('-f\nbar')

So the file contains:

-f
bar

In other words, each of the file lines corresponds to one 'word' (blank separated) in the command line. --option1=foo is one word. --option1 foo is interpreted just as though it was quoted in the command line,eg. prog.py '--option1 foo' '--option2 1234'

The https://docs.python.org/dev/library/argparse.html#argparse.ArgumentParser.convert_arg_line_to_args has a custom function that will split lines on spaces. Experiment with that if you want to stick with the argument file.

---

import argparse

with open('args.txt', 'w') as fp:
    fp.write('--option1 foo\n--option2 1234')  # error
    # but works with modifed 'convert...'
    #fp.write('--option1=foo\n--option2=1234')  # works
    #fp.write('--option1\nfoo\n--option2\n1234') # works

def convert_arg_line_to_args(arg_line):
    for arg in arg_line.split():
        if not arg.strip():
            continue
        yield arg
"""
default line converter:
def convert_arg_line_to_args(self, arg_line):
    return [arg_line]
"""

def go():
   parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
   parser.convert_arg_line_to_args = convert_arg_line_to_args
   parser.add_argument("--option1")
   parser.add_argument("--option2", type=int, required=True)
   args = parser.parse_args(['@args.txt'])
   print args

if __name__ == "__main__":
    go()

---

@toes suggested using shlex to parse the file. shlex has a nice feature in that it strips of unnecessary quotes.

shlex can be used to split individual lines of the file.

def sh_split(arg_line):
    for arg in shlex.split(arg_line):
        yield arg
parser.convert_arg_line_to_args = sh_split

Or it can replace the whole @file read method (_read_args_from_files)- this should function the same as @toes answer, except that the @file string can be anywhere in the commandline (or even be repeated).

def at_read_fn(arg_strings):
    # expand arguments referencing files
    new_arg_strings = []
    for arg_string in arg_strings:
        if not arg_string or not arg_string.startswith('@'):
            new_arg_strings.append(arg_string)
        else:
            with open(arg_string[1:]) as args_file:
                arg_strings = shlex.split(args_file.read())
                new_arg_strings.extend(arg_strings)
    return new_arg_strings
parser._read_args_from_files = at_read_fn

Obviously a cleaner production version would modify these methods in an ArgumentParser subclass.