Paramiko and exec_command - killing remote process?

Question

I'm using Paramiko to tail -f a file on a remote server.

Previously, we were running this via ssh -t, but that proved flaky, and the -t caused issues with our remote scheduling system.

My question is how to kill tail when the script catches a SIGINT?

My script (based on Long-running ssh commands in python paramiko module (and how to end them))

#!/usr/bin/env python2
import paramiko
import select

client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username='victorhooi', password='blahblah')
transport = client.get_transport()
channel = transport.open_session()

channel.exec_command("tail -f /home/victorhooi/macbeth.txt")
while True:
    try:
        rl, wl, xl = select.select([channel],[],[],0.0)
        if len(rl) > 0:
            # Must be stdout
            print channel.recv(1024)
    except KeyboardInterrupt:
        print("Caught control-C")
        client.close()
        channel.close()
        exit(0)

The script catches my Ctrl-C successfully, and ends. However, it leaves the tail -f process running on the remote system,.

Neither client.close() nor channel.close() seem to terminate it.

What command can I issue in the except block to kill it?

The remote server is running Solaris 10.

Answer 1

While not the most efficient method, this should work. After you CTRL+C; In the KeyboardInterrupt handler you could exec_command("killall -u %s tail" % uname) like so:

#!/usr/bin/env python2

import paramiko
import select

import time
ltime = time.time()

# Or use random:
# import random
# ltime = random.randint(0, 500)

uname = "victorhooi"
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username=uname, password='blahblah')
transport = client.get_transport()
channel = transport.open_session()

channel.exec_command("tail -%df /home/victorhooi/macbeth.txt" % ltime)
while True:
    try:
        rl, wl, xl = select.select([channel],[],[],0.0)
        if len(rl) > 0:
            # Must be stdout
            print channel.recv(1024)
    except KeyboardInterrupt:
        print("Caught control-C")
        channel.close()
        try:
            # open new socket and kill the proc..
            client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))
        except:
            pass
    
        client.close()
        exit(0)

This would kill any open processes named tail. That may cause issues though if you have tails open that you dont want to close, if thats the case you could grep a ps, get the pid and kill -9 it.

First, set tail to read n lines from end of file before following. set n to a unique nuber like time.time(), since tail doesn't care if that number is larger then the number of lines in the file, the large number from time.time()shouldnt cause issues and will be unique. Then grep for that unique number in the ps:

   client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))

Answer 2

There is one way to do this. It works like on the shell

ssh -t commandname

The option -t is opening a pseudo pty to help ssh to track how long this process should last. the same can be done via pormiko via

channel.get_pty()

prior to execute_command(...). This will not open a shell like it does with channel.invoke_shell(), it just requests such a pseudo interface to tie all processes to. The effect can also be seen if ps aux is issued on the remote machine, the process is now attached to the sshd with a ptxXY interface.

Answer 3

You should use ssh keepalives... the problem you have is that the remote shell has no way of knowing (by default) that your ssh session was killed. Keepalives will enable the remote shell to detect that you killed the session

client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username='victorhooi', password='blahblah')
transport = client.get_transport()
transport.set_keepalive(1)   # <------------------------------
# ... carry on as usual...

Set the keepalive value as low as you like (even 1 second)... after several seconds, the remote shell will see that the ssh login died, and it will terminate any processes that were spawned by it.

Answer 4

I just hit this issue and wasn't in a position to issue a pkill to close the process at the end.

A better solution is to change the command you are running to:

tail -f /path/to/file & { read ; kill %1; }

This will let you run your tail command for as long as you need. As soon as you send a newline to the remote process the kill %1 will execute and stop the tail command you backgrounded. (for reference: %1 is a jobspec and used to describe the first process that has been backgrounded in your session, ie the tail command)

Answer 5

Here's a way to obtain the remote process ID:

def execute(channel, command):
    command = 'echo $$; exec ' + command
    stdin, stdout, stderr = channel.exec_command(command)
    pid = int(stdout.readline())
    return pid, stdin, stdout, stderr

And here's how to use it (replace ... with the bits in the original question):

pid, _, _, _ = execute(channel, "tail -f /home/victorhooi/macbeth.txt")
while True:
    try:
        # ...
    except KeyboardInterrupt:
        client.exec_command("kill %d" % pid)
        # ...

Answer 6

Specifically for 'tail' you could use the --pid=PID argument and let tail take care of it:

  --pid=PID  with -f, terminate after process ID, PID dies

Answer 7

You can use get_pty as described in https://stackoverflow.com/a/38883662/565212.

E.g. scenario - When to call client/channel.close():
Step1: Execute a remote command that writes to the log file.
Step2: Spawn a thread that executes the tail command and blocks in the readline loop
Step3: In main thread, when the command returns, you know there will be no more logs, kill the tail thread.

Answer 8

Had the same problem with ssh -t. There is a library called closer - it runs a remote process via ssh and closes automatically for you. Check it out.