How to script multiple ssh and scp commands to various systems

Question

I would like to script a sequence of commands involving multiple ssh and scp calls. On a daily basis, I find myself manually performing this task:

From LOCAL system, ssh to SYSTEM1
mkdir /tmp/data on SYSTEM1
from SYSTEM1, ssh to SYSTEM2
mkdir /tmp/data on SYSTEM2
from SYSTEM2, SSH to SYSTEM3

scp files from SYSTEM3:/data to SYSTEM2:/tmp/data
exit to SYSTEM2
scp files from SYSTEM2:/data and SYSTEM2:/tmp/data to SYSTEM1:/tmp/data
rm -fr SYSTEM2:/tmp/data
exit to SYSTEM1
scp files from SYSTEM1:/data and SYSTEM1:/tmp/data to LOCAL:/data
rm -fr SYSTEM1:/tmp/data

I do this process at LEAST once a day and it takes approximately 5-10 minutes going between the different systems and then cleaning up afterwards. I would really like to automate this in a bash script but my amateur attempts so far have been unsuccessful. As you might suspect, the systems communication is constrained, meaning LOCAL can only see System1, System2 can only see System1 and System3, system3 can only see system2, etc. You get the idea. What is the best way to do this? Additionally, System1 is a hub for many other systems so SYSTEM2 must be indicated by the user (System3 will always have the same relative IP/hostname compared to any SYSTEM2).

I tried just putting the commands in the proper order in a shell script and then manually typing in the passwords when prompted (which would already be a huge gain in efficiency) but either the method doesn't work or my execution of the script is wrong. Additionally, I would want to have a command line argument for the script that would take a pattern for which 'system2' to connect to, a pattern for the data to copy, and a target location for the data on the local system.

Such as

./grab_data system2 *05-14* ~/grabbed-data

I did some searching and I think my next step would be to have scripts on each system that perform the local tasks, and then execute the scripts via ssh commands from the respective remote system. Is there a better way? What commands should I look at using and what would be the general approach to this automating this sort of nested ssh and scp problem?

I realize my description may be a bit convoluted so please ask for clarification on any area that I did not properly describe.

Thanks.

Answer 1

You can simplify this process a lot by tunneling ssh connections over other ssh connections (see this previous answer). The way I'd do it is to create an .ssh/config file on the LOCAL system with the following entries:

Host SYSTEM3
        ProxyCommand    ssh -e none SYSTEM2 exec /usr/bin/nc %h %p 2>/dev/null
        HostName        SYSTEM3.full.domain
        User            system3user

Host SYSTEM2
        ProxyCommand    ssh -e none SYSTEM1 exec /usr/bin/nc %h %p 2>/dev/null
        HostName        SYSTEM2.full.domain
        User            system2user

Host SYSTEM1
        HostName        SYSTEM1.full.domain
        User            system1user

(That's assuming both intermediate hosts have netcat installed as /usr/bin/nc -- if not, you may have to find/install some equivalent way of gatewaying stdin&stdout into a TCP session.)

With this set up, you can use scp SYSTEM3:/data /data on LOCAL, and it'll automatically tunnel through SYSTEM1 and SYSTEM2 (and ask for the passwords for the three SYSTEMn's in order -- this can be a little confusing, especially if you mistype one).

Answer 2

If you're connecting to multiple systems, and especially if you have to forward connections through intermediate hosts, you will want to use public key authentication with ssh-agent forwarding enabled. That way, you only have to authenticate once.

Scripted SSH with agent forwarding may suffice if all you need to do is check the exit status from your remote commands, but if you're going to do anything complex you might be better off using expect or expect-lite to drive the SSH/SCP sessions in a more flexible way. Expect in particular is designed to be a robust replacement for interactive sessions.

If you stick with shell scripting, and your filenames change a lot, you can always create a wrapper around SSH or SCP like so:

# usage: my_ssh [remote_host] [command_line]
# returns: exit status of remote command, or 255 on SSH error
my_ssh () {
    local host="$1"
    shift
    ssh -A "$host" "$@"
}

Between ssh-agent and the wrapper function, you should have a reasonable starting point for your own efforts.

Answer 3

Another way could be to use rsync, which tomatically creates any needed directories and, if you want, removes the copied source files.

In your case, you could work with the commands

home:~$ ssh system1
system1:~$ ssh system2
system2:~$ rsync -aPSHiv system3:/data /tmp/data
system2:~$ exit
system1:~$ rsync -aPSHiv --remove-source-files system2:/tmp/data /tmp/data
system1:~$ rsync -aPSHiv system2:/data /tmp/data
system1:~$ exit
home:~$ rsync -aPSHiv --remove-source-files system1:/tmp/data /tmp/data
home:~$ rsync -aPSHiv system1:/data /data

If you combine this with Gordon's approach, you can even reduce that to

home:~$ rsync -aPSHiv system1:/data/ system2:/data/ system3:/data/ /data/

Note that rsync makes a difference between ...data and ...data/ - the former means the directory and its contents, the latter just the contents. If you mix them up, you might end up with a directory data in another directory data.

Besides, you simplify things if you work with public SSH keys instead of passwords.