Paramiko SSH Tunnel Shutdown Issue

Question

I'm working on a python script to query a few remote databases over an established ssh tunnel every so often. I'm fairly familiar with the paramiko library, so that was my choice of route. I'd prefer to keep this in complete python so I can use paramiko to deal with key issues, as well as uses python to start, control, and shutdown the ssh tunnels.

There have been a few related questions around here about this topic, but most of them seemed incomplete in answers. My solution below is a hacked together of the solutions I've found so far.

Now for the problem: I'm able to create the first tunnel quite easily (in a separate thread) and do my DB/python stuff, but when attempting to close the tunnel the localhost won't release the local port I binded to. Below, I've included my source and the relevant netstat data through each step of the process.

#!/usr/bin/python

import select
import SocketServer
import sys
import paramiko
from threading import Thread
import time



class ForwardServer(SocketServer.ThreadingTCPServer):
    daemon_threads = True
    allow_reuse_address = True

class Handler (SocketServer.BaseRequestHandler):
    def handle(self):
        try:
            chan = self.ssh_transport.open_channel('direct-tcpip', (self.chain_host, self.chain_port), self.request.getpeername())
        except Exception, e:
            print('Incoming request to %s:%d failed: %s' % (self.chain_host, self.chain_port, repr(e)))
            return
        if chan is None:
            print('Incoming request to %s:%d was rejected by the SSH server.' % (self.chain_host, self.chain_port))
            return
        print('Connected!  Tunnel open %r -> %r -> %r' % (self.request.getpeername(), chan.getpeername(), (self.chain_host, self.chain_port)))
        while True:
            r, w, x = select.select([self.request, chan], [], [])
            if self.request in r:
                data = self.request.recv(1024)
                if len(data) == 0:
                    break
                chan.send(data)
            if chan in r:
                data = chan.recv(1024)
                if len(data) == 0:
                    break
                self.request.send(data)
        chan.close()
        self.request.close()
        print('Tunnel closed from %r' % (self.request.getpeername(),))

class DBTunnel():

    def __init__(self,ip):
        self.c = paramiko.SSHClient()
        self.c.load_system_host_keys()
        self.c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.c.connect(ip, username='someuser')
        self.trans = self.c.get_transport()

    def startTunnel(self):
        class SubHandler(Handler):
            chain_host = '127.0.0.1'
            chain_port = 5432
            ssh_transport = self.c.get_transport()
        def ThreadTunnel():
            global t
            t = ForwardServer(('', 3333), SubHandler)
            t.serve_forever()
        Thread(target=ThreadTunnel).start()

    def stopTunnel(self):
        t.shutdown()
        self.trans.close()
        self.c.close()

Although I will end up using a stopTunnel() type method, I've realize that code isn't entirely correct, but more so an experimentation of trying to get the tunnel to shutdown properly and test my results.

When I first call create the DBTunnel object and call startTunnel(), netstat yields the following:

tcp4       0      0 *.3333                 *.*                    LISTEN
tcp4       0      0 MYIP.36316      REMOTE_HOST.22                ESTABLISHED
tcp4       0      0 127.0.0.1.5432         *.*                    LISTEN

Once I call stopTunnel(), or even delete the DBTunnel object itself..I'm left with this connection until I exit python all together, and what I assume to be the garbage collector takes care of it:

tcp4       0      0 *.3333                 *.*                    LISTEN

It would be nice to figure out why this open socket is hanging around independent of the DBConnect object, and how to close it properly from within my script. If I try and bind a different connection to different IP using the same local port before completely exiting python (time_wait is not the issue), then I get the infamous bind err 48 address in use. Thanks in advance :)

Answer 1

It appears the SocketServer's shutdown method isn't properly shutting down/closing the socket. With the below changes in my code, I retain access to the SocketServer object and access the socket directly to close it. Note that socket.close() works in my case, but others might be interested in socket.shutdown() followed by a socket.close() if other resources are accessing that socket.

[Ref: socket.shutdown vs socket.close

def ThreadTunnel():
    self.t = ForwardServer(('127.0.0.1', 3333), SubHandler)
    self.t.serve_forever()
Thread(target=ThreadTunnel).start()

def stopTunnel(self):
    self.t.shutdown()
    self.trans.close()
    self.c.close()
    self.t.socket.close()

Answer 2

Note that you don't have do the Subhandler hack as shown in the demo code. The comment is wrong. Handlers do have access to their Server's data. Inside a handler you can use self.server.instance_data.

If you use the following code, in your Handler, you would use

• self.server.chain_host
• self.server.chain_port
• self.server.ssh_transport

---

class ForwardServer(SocketServer.ThreadingTCPServer):
    daemon_threads = True
    allow_reuse_address = True

    def __init__(
          self, connection, handler, chain_host, chain_port, ssh_transport):
        SocketServer.ThreadingTCPServer.__init__(self, connection, handler)
        self.chain_host = chain_host
        self.chain_port = chain_port
        self.ssh_transport = ssh_transport
...

server = ForwardServer(('', local_port), Handler, 
                       remote_host, remote_port, transport)
server.serve_forever()

Answer 3

You may want to add some synchronization between the spawned thread and the caller so that you don't try to use the tunnel before it is ready. Something like:

    from threading import Event   
    def startTunnel(self):
        class SubHandler(Handler):
            chain_host = '127.0.0.1'
            chain_port = 5432
            ssh_transport = self.c.get_transport()
        mysignal = Event()
        mysignal.clear()
        def ThreadTunnel():
            global t
            t = ForwardServer(('', 3333), SubHandler)
            mysignal.set() 
            t.serve_forever()
        Thread(target=ThreadTunnel).start()
        mysignal.wait()