Looping over pairs of values in bash

Question

I have 10 text files and I want to paste each file with its pair, such that I have 5 total files.

I tried the following:

for i in 4_1 5_         


        

Answer 1

There is a common pattern where you have pairs of files, where one name of the pair can be easily derived from the other. If the file you know the name of is X and the other file is Y, you have the following common use cases.

• For renaming, Y is X with an extension removed and/or a date stamp added.
• For transcoding, Y is X with a different extension and perhaps a different directory.
• For many data analysis tasks, X and Y share some parts of the file name, but have different parameters or extensions.

All of these lend themselves to the same rough code skeleton.

for x in path/to/base*.ext; do
    dir=${x%/*}   # Trim trailing file name, keep dir
    base=${x##*/} # Trim any leading directory

    # In this case, $y has a different subdirectory and a different extension
    y=${dir%/to}/from/${base%.ext}.newext

    # Maybe check if y exists?  Or doesn't exist?
    if [ -e "$y" ]; then
        echo "$0: $y already exists -- skipping" >&2
        continue
    fi

    mv or ffmpeg or awk or whatever "$x" and "$y"
done

The key here is the observation that y can be derived from x with some simple variable substitutions. So you loop over the x values, and figure out the corresponding y value inside the loop.

Here, we have used the shell's built-in ${variable#prefix} and ${variable%suffix} operators to return the variable's value with any leading prefix or trailing suffix, respectively, trimmed off. (There is also ## and %% to match the longest, instead of the shortest, possible match. The expression after # or % is a regular shell glob pattern.) These should usually be all you need, although you frequently see sed or awk scripts even for this trivial job (where really you should usually try to avoid an external process), as well as of course for more demanding transformations.

If you need to loop over x files scattered across different directories, maybe the loop should start with something like

 find dir1 dir2 etc/and/so/forth -type f -name 'x-files*.ext' -print |
 while IFS='' read -r x; do
     :

A commonly seen problem in similar questions is answers which fail to quote $x and $y correctly. Generally, any variable containing a file name should always be in double quotes.

Where X and Y are unrelated, a common solution is to loop over a here document containing the mapping:

while read -r x y; do
    : stuff with "$x" and "$y"
done <<'____HERE'
    first_x_value  first_y_value
    another_x      corresponding_y
    random         surprise
____HERE