Sending strings between to Python Scripts using subprocess PIPEs

Question

I want to open a Python script using subprocess in my main python program. I want these two programs to be able to chat with one another as they are both running so I can monitor the activity in the slave script, i.e. I need them to send strings between each other.

The main program will have a function similar to this that will communicate with and monitor the slave script:

Script 1

import subprocess
import pickle
import sys
import time
import os

def communicate(clock_speed, channel_number, frequency):
    p = subprocess.Popen(['C:\\Python27\\pythonw','test.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    data = pickle.dumps([clock_speed, channel_number, frequency]).replace("\n", "\\()")
    print data
    p.stdin.write("Start\n")
    print p.stdout.read()
    p.stdin.write(data + "\n")
    p.poll()
    print p.stdout.readline()
    print "return:" + p.stdout.readline()
    #p.kill()

if __name__ == '__main__':
    print "GO"
    communicate(clock_speed = 400, channel_number = 0, frequency = 5*1e6)

The test.py script looks similar to this:

Script 2

import ctypes
import pickle
import time
import sys

start = raw_input("")
sys.stdout.write("Ready For Data")
data = raw_input("")
data = pickle.loads(data.replace("\\()", "\n"))
sys.stdout.write(str(data))
###BUNCH OF OTHER STUFF###

What I want these scripts to do is the following:

1. Script 1 to open Script 2 using Popen
2. Script 1 sends the string "Start\n"
3. Script 2 reads this string and sends the string "Ready For Data"
4. Script 1 reads this string and sends the pickled data to Script 2
5. Then whatever...

The main question is how to do parts 2-4. Then the rest of the communication between the two scripts should follow. As of now, I have only been able to read the strings from Script 2 after it has been terminated.

Any help is greatly appreciated.

UPDATE:

Script 1 must be run using 32-bit Python, while Script 2 must be run using 64-bit Python.

Answer 1

The problem with pipes is that if you call a read operation and there is nothing to read, your code is stalled until the other party writes something for you to read. Also if you write too much, your next write operation might block until the other party reads something out of the pipe and frees it.

There are "non-blocking calls" you can make, that will return an error in these cases instead of blocking, but your application will still need to deal with the errors sensibly.

In any case, you need to set up some kind of protocol. Think of HTTP, or any other protocol you know well: there are requests and responses, and while you are reading either of the two the protocol always tells you if there is something else you need to read or not. That way you can always make an informed decision on whether to wait for more data or not.

Here is an example that works. It works because there is the following protocol:

• p1 sends a single line, ending with '\n';
• p2 does the same;
• p1 sends another line;
• p2 does the same;
• both are happy and exit.

In order to write a line to the pipe (on either side) and make sure it gets onto the pipe, I call write() and then flush().

In order to read a single line from the pipe (on either side) but not a single byte more, thus blocking my code until the line is ready and no longer than that, I use readline().

There are other calls you could make and other protocols, including ready-made ones, but the single-line protocol works well for simple things and for a demo like this.

p1.py:

import subprocess

p = subprocess.Popen(['python', 'p2.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p.stdin.write("Hello\n")
p.stdin.flush()
print 'got', p.stdout.readline().strip()
p.stdin.write("How are you?\n")
p.stdin.flush()
print 'got', p.stdout.readline().strip()

p2.py:

import sys

data = sys.stdin.readline()
sys.stdout.write("Hm.\n")
sys.stdout.flush()
data = sys.stdin.readline()
sys.stdout.write("Whatever.\n")
sys.stdout.flush()

Answer 2

I also had a problem similar to this, where there was no way to send general Python objects between different processes without running into the problem of knowing either when the other side hasn't sent an object or is closed. Also trying to use multiprocessing.Queue usually means that the process needs to have been started by the current process which is not always the case when two processes want to collaborate.

To combat this I use the picklepipe module, which defines a generic object serialization pipe interface as well as a pipe that uses the pickle protocol called the PicklePipe (also one that uses the marshal protocol called MarshalPipe). It can send more than just strings, it can send any pickleable object to it's peer.

The pipes are even selectable, meaning they can be used by the selectors module (or selectors2, selectors34) as file objects when a new object is ready to be received. This makes waiting for many different pipes to be ready very efficient.

Supports Python 2.7+ (and probably 2.6) and all major platforms. Can even send objects between two different versions of Python! Check out the project documentation or view the source on Github.

Disclosure: I am the author of picklepipe. I would love to hear your feedback. :)