Paramiko Sessions Closes Transport in the Child Process

问题

We are using paramiko to make a connection library which heavily uses its get_pty or invoke_shell features. Our library uses these channels for interacting with the target device.

But whenever we use multiprocessing library, we are not able to use paramiko connection handles in the child process. The transport gets closed in the child process.

Is there a way to tell paramiko not to close the connection/channel at fork.

This is the sample program for reproducing the problem

from paramiko import SSHClient, AutoAddPolicy
from multiprocessing import Process
import logging
log = logging.getLogger("paramiko.transport").setLevel(1)

client = SSHClient()

client.set_missing_host_key_policy(AutoAddPolicy())

client.connect(hostname="localhost")

def simple_work(handle):
    print("==== ENTERED CHILD PROCESS =====")
    stdin, stdout, stderr = handle.exec_command("ifconfig")
    print(stdout.read())
    print("==== EXITING CHILD PROCESS =====")

p = Process(target=simple_work, args=(client,))
p.start()
p.join(2)
print("==== MAIN PROCESS AFTER JOIN =====")
stdin, stdout, stderr = client.exec_command("ls")
print(stdout.read())

and this is the error

==== ENTERED CHILD PROCESS =====
Success for unrequested channel! [??]
==== MAIN PROCESS AFTER JOIN =====
Traceback (most recent call last):
  File "repro.py", line 22, in <module>
    stdin, stdout, stderr = client.exec_command("ls")
  File "/Users/vivejha/Projects/cisco/lib/python3.4/site-packages/paramiko/client.py", line 401, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/Users/vivejha/Projects/cisco/lib/python3.4/site-packages/paramiko/transport.py", line 702, in open_session
    timeout=timeout)
  File "/Users/vivejha/Projects/cisco/lib/python3.4/site-packages/paramiko/transport.py", line 823, in open_channel
    raise e
paramiko.ssh_exception.SSHException: Unable to open channel.

Few important things to note

If I try to access the client in the child process. First of all it does't work at all.
Secondly, the handle in the main process also dies out surprisingly. I don't how this child-to-parent communication is facilitated and why.
And the biggest problem is program hangs in the end, exception is fine but hangs are least expected.
If I don't use the client in the child process, and do some work other work then the client in the parent process is not impacted and works as usual.

NOTE: There is something called atfork inside the transport.py which claims to control this behaviour. But surprisingly even commenting the code in that method has no impact. Also there are no references to atfork in the entire codebase of paramiko.

PS: I am using latest paramiko and this program was run on Mac

回答1:

It is just a fundamental problem when sockets are involved with fork. Both processes share the same socket but only one can use it. Just imagine that two different processes are managing one socket. They both are in different states e.g. one might send and receive data to the remote side while the other one is in a totally different crypto state. Just think about nonces/initialization vectors, they'll just be invalid after forking when both processes diverge.

The solution to your problem is obviously to switch from MultiProcessing to MultiThreading. This way you only have one ssh connection that is shared across all threads. If you really want to use fork you'll have to fork with creating one new connection per fork.

see transport.py

def atfork(self):
    """
    Terminate this Transport without closing the session.  On posix
    systems, if a Transport is open during process forking, both parent
    and child will share the underlying socket, but only one process can
    use the connection (without corrupting the session).  Use this method
    to clean up a Transport object without disrupting the other process.

In paramiko log you'll see that your parent process receivs a SSH_DISCONNECT_MSG from the remote side with the error: Packet corrupt. Most likely due to the parent being in a different crypto state and sending a packet that the server could not understand.

DEBUG:lala:==== ENTERED CHILD PROCESS =====
DEBUG:lala:<paramiko.SSHClient object at 0xb74bf1ac>
DEBUG:lala:<paramiko.Transport at 0xb6fed82cL (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>
DEBUG:paramiko.transport:[chan 1] Max packet in: 34816 bytes
WARNING:paramiko.transport:Success for unrequested channel! [??]


DEBUG:lala:==== MAIN PROCESS AFTER JOIN =====
WARNING:lala:<socket._socketobject object at 0xb706ef7c>
DEBUG:paramiko.transport:[chan 1] Max packet in: 34816 bytes
INFO:paramiko.transport:Disconnect (code 2): Packet corrupt

Here's a basic MultiThreading example using concurrent.futures:

from concurrent.futures import ThreadPoolExecutor

def simple_work(handle):
    print("==== ENTERED CHILD PROCESS =====")
    stdin, stdout, stderr = handle.exec_command("whoami")
    print(stdout.read())
    print("==== EXITING CHILD PROCESS =====")

with ThreadPoolExecutor(max_workers=2) as executor:
    future = executor.submit(simple_work, client)
    print(future.result())

print("==== MAIN PROCESS AFTER JOIN =====")
stdin, stdout, stderr = client.exec_command("echo AFTER && whoami")
print(stdout.read())

Also note that in most cases you do not even need to introduce extra threading. Paramiko exec_command alread spawns a new thread and will not block until you try to read from any pseudofile stdout,stderr. That means, you could as well just execute a few commands and read from stdout lateron. But keep in mind that paramiko might stall due to buffers running full.

来源：https://stackoverflow.com/questions/33671328/paramiko-sessions-closes-transport-in-the-child-process

标签

python

paramiko