get variables substituted when I cat a file

Question

Is it possible, in a clean way to get the variable values when I cat a file, instead of the variable names, as written in the file. It's hard to explain, but here goes a simple example:

$ cat <<EOF
$HOME
EOF
/home/myself

cat returns /home/myself because it is already expanded by the shell.

$ echo \$HOME >/tmp/home
$ cat /tmp/home
$HOME

cat simply reads the file, I want $HOME to be expanded here somehow by cat, because the file will contain variable names (not like HOME=/home/myself)

My question is if this is possible somehow, otherwise I will have to write some dirty code.

EDIT: they are big xml files containing

<checkbox active="$value">

true or false

Answer 1

This would be trivial to attempt in Python, and see if that works for you. You could use the re.sub function to replace all occurrences of some pattern like "\$\w+" by calling a function which does the transformation (rather than a specific string to replace it with). And for the replacement function you could use os.getenv(), which of course takes a variable name and returns its value.

Edit: Here's a complete Python script that does the above:

#!/usr/bin/python

import fileinput
import os
import re

def transform(match):
    return os.getenv(match.group(1)) # replace the "capture" to omit $

for line in fileinput.input(): # reads from stdin or from a file in argv
    print re.sub('\$(\w+)', transform, line), # comma to omit newline

Answer 2

The obvious way to do this has a lot of problems:

# This will fail with certain inputs.  HTML will certainly be a problem
# as the '<' and '>' characters will be interpreted as file redirects
$ while read r; do eval echo $r; done < input

The following perl should handle the problem fairly well for simple inputs.

$ perl -pwe 'while(($k,$v) = each %ENV ) { s/\${?$k}?/$v/ }' input

But it doesn't do anything with constructs like ${FOO-bar}. If you need to handle such constructs, it might be sufficient to escape all of the shell meta-characters and do the while/read loop:

$ sed -e 's/\([<>&|();]\)/\\\1/g' input | while read -r l; do eval echo "$l"; done

Note that this is neither robust nor secure. Consider what happens on input like:

\; rm -rf /

I said "consider". Do not test that. The sed will insert a backslash before the semicolon, the eval will then get the string "\\;" which will be interpreted as a single backslash followed by a semi-colon which termintates the echo, and the rm -rf will be executed. Given the insecurity of evaling unknown input, it would probably be safer to stick with something like perl and explicitly replace the desired sh constructs. Something like:

$ perl -pwe 'while(($k,$v) = each %ENV ) { s/\${?$k}?/$v/ }; 
    s/\${[^-]*-([^}]*)}/$1/g' input

This one has problems with input like ${FOO=some-text}. In order to reliably get all of the sh constructs (${word:rhs} where the ':' can be any of '-', '?', '=', '+', '%', '#' or any of the same with a colon prepended (or a lot of other symbols if you allow non-posix sh syntax!)) you would have to construct a fairly elaborate set of comparisons.

Answer 3

cat copies its inputs unchanged to its outputs - at least in its original (1st Edition UNIX) form. It didn't have any options to start with. Then BSD added a bunch, and the original UNIX team objected: 'cat came back from Berkeley waving flags' (see: 1 - passim). It should not be used to edit files - that is not its purpose. (I found a reference to the article in the BSD (Mac OS X) man page for cat: Rob Pike, "UNIX Style, or cat -v Considered Harmful", USENIX Summer Conference Proceedings, 1983. See also http://quotes.cat-v.org/programming/)

So, you need something other than cat to do the job. I'd recommend Perl or Python; either can do it pretty easily. Alternatively, consider sed, or perhaps awk.

#!/usr/bin/env perl
use strict;
use warnings;
while (<>)
{
    foreach my $key (keys %ENV)
    {
        s/\$$key\b/$ENV{$key}/g;  # $envvar
        s/\${$key}/$ENV{$key}/g;  # ${envvar}
    }
    print;
}

This loops through the input line, looking for each environment variable in turn. The alternative mechanism is to look for possible variables and do the relevant substitution. This turns out to be a little tricky, but doable:

#!/usr/bin/env perl
use strict;
use warnings;
while (<>)
{
    while (m/\$((\w+))/ || m/\$({(\w+)})/)
    {
        my $key = $2;
        my $var = $1;
        s/\$$var/$ENV{$key}/g if defined $ENV{$key};
    }
    print;
}

When I included the literal $ in the captures, the substitute operation did not work correctly.