In Python, how to write a string to a file on a remote machine?

Question

On Machine1, I have a Python2.7 script that computes a big (up to 10MB) binary string in RAM that I\'d like to write to a disk file on Machine2, which is a remote machine.

Answer 1

You open a new SSH process to Machine2 using subprocess.Popen and then you write your data to its STDIN.

import subprocess

cmd = ['ssh', 'user@machine2',
       'mkdir -p output/dir; cat - > output/dir/file.dat']

p = subprocess.Popen(cmd, stdin=subprocess.PIPE)

your_inmem_data = 'foobarbaz\0' * 1024 * 1024

for chunk_ix in range(0, len(your_inmem_data), 1024):
    chunk = your_inmem_data[chunk_ix:chunk_ix + 1024]
    p.stdin.write(chunk)

I've just verified that it works as advertised and copies all of the 10485760 dummy bytes.

P.S. A potentially cleaner/more elegant solution would be to have the Python program write its output to sys.stdout instead and do the piping to ssh externally:

$ python process.py | ssh