how to “source” file into python script

Question

I have a text file /etc/default/foo which contains one line:

FOO=\"/path/to/foo\"

In my python script, I need to reference the

Answer 1

The Solution

Here is my approach: parse the bash file myself and process only variable assignment lines such as:

FOO="/path/to/foo"

Here is the code:

import shlex

def parse_shell_var(line):
    """
    Parse such lines as:
        FOO="My variable foo"

    :return: a tuple of var name and var value, such as
        ('FOO', 'My variable foo')
    """
    return shlex.split(line, posix=True)[0].split('=', 1)

if __name__ == '__main__':
    with open('shell_vars.sh') as f:
        shell_vars = dict(parse_shell_var(line) for line in f if '=' in line)
    print(shell_vars)

How It Works

Take a look at this snippet:

        shell_vars = dict(parse_shell_var(line) for line in f if '=' in line)

This line iterates through the lines in the shell script, only process those lines that has the equal sign (not a fool-proof way to detect variable assignment, but the simplest). Next, run those lines into the function parse_shell_var which uses shlex.split to correctly handle the quotes (or the lack thereof). Finally, the pieces are assembled into a dictionary. The output of this script is:

{'MOO': '/dont/have/a/cow', 'FOO': 'my variable foo', 'BAR': 'My variable bar'}

Here is the contents of shell_vars.sh:

FOO='my variable foo'
BAR="My variable bar"
MOO=/dont/have/a/cow
echo $FOO

Discussion

This approach has a couple of advantages:

• It does not execute the shell (either in bash or in Python), which avoids any side-effect
• Consequently, it is safe to use, even if the origin of the shell script is unknown
• It correctly handles values with or without quotes

This approach is not perfect, it has a few limitations:

• The method of detecting variable assignment (by looking for the presence of the equal sign) is primitive and not accurate. There are ways to better detect these lines but that is the topic for another day

• It does not correctly parse values which are built upon other variables or commands. That means, it will fail for lines such as:

  FOO=$BAR
  FOO=$(pwd)
  

Answer 2

Just to give a different approach, note that if your original file is setup as

export FOO=/path/to/foo

You can do source /etc/default/foo; python myprogram.py (or . /etc/default/foo; python myprogram.py) and within myprogram.py all the values that were exported in the sourced' file are visible in os.environ, e.g

import os
os.environ["FOO"] 

Answer 3

There are a couple ways to do this sort of thing.

• You can indeed import the file as a module, as long as the data it contains corresponds to python's syntax. But either the file in question is a .py in the same directory as your script, either you're to use imp (or importlib, depending on your version) like here.

• Another solution (that has my preference) can be to use a data format that any python library can parse (JSON comes to my mind as an example).

/etc/default/foo :

{"FOO":"path/to/foo"}

And in your python code :

import json

with open('/etc/default/foo') as file:
    data = json.load(file)
    FOO = data["FOO"]
    ## ...
    file.close()

This way, you don't risk to execute some uncertain code...

You have the choice, depending on what you prefer. If your data file is auto-generated by some script, it might be easier to keep a simple syntax like FOO="path/to/foo" and use imp.

Hope that it helps !

Answer 4

You could use execfile:

execfile("/etc/default/foo")

But please be aware that this will evaluate the contents of the file as is into your program source. It is potential security hazard unless you can fully trust the source.

It also means that the file needs to be valid python syntax (your given example file is).

Answer 5

Same answer as @jil however, that answer is specific to some historical version of Python.

In modern Python (3.x):

exec(open('filename').read())

replaces execfile('filename') from 2.x

Answer 6

If you know for certain that it only contains VAR="QUOTED STRING" style variables, like this:

FOO="some value"

Then you can just do this:

>>> with open('foo.sysconfig') as fd:
...   exec(fd.read())

Which gets you:

>>> FOO
'some value'

(This is effectively the same thing as the execfile() solution suggested in the other answer.)

This method has substantial security implications; if instead of FOO="some value" your file contained:

os.system("rm -rf /")

Then you would be In Trouble.

Alternatively, you can do this:

>>> with open('foo.sysconfig') as fd:
...   settings = {var: shlex.split(value) for var, value in [line.split('=', 1) for line in fd]}

Which gets you a dictionary settings that has:

>>> settings
{'FOO': ['some value']}

That settings = {...} line is using a dictionary comprehension. You could accomplish the same thing in a few more lines with a for loop and so forth.

And of course if the file contains shell-style variable expansion like ${somevar:-value_if_not_set} then this isn't going to work (unless you write your very own shell style variable parser).