python os.mkfifo() for Windows

Question

Short version (if you can answer the short version it does the job for me, the rest is mainly for the benefit of other people with a similar task):

In python in Wind

Answer 1

os.pipe() returns an anonymous pipe, or a named pipe on Windows, which is very lightweight and efficient.

TCP sockets (as suggested by user1495323) are more heavyweight: you can see them with netstat for example, and each one requires a port number, and the number of available ports is limited to 64k per peer (e.g. 64k from localhost to localhost).

On the other hand, named pipes (on Windows) are limited because:

• You can't use select() for nonblocking I/O on Windows, because they're not sockets.
• There's no apparent way to read() with a timeout, and
• Even making them non-blocking is difficult.

And sockets can be wrapped in Python-compatible filehandles using makefile(), which allows them to be used to redirect stdout or stderr. This makes this an attractive option for some use cases, such as sending stdout from one thread to another.

A socket can be constructed with an automatically-assigned port number like this (based on the excellent Python socket HOWTO):

with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as input_socket:
    # Avoid socket exhaustion by setting SO_REUSEADDR :
    input_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    # localhost doesn't work if the definition is missing from the hosts file,
    # and 127.0.0.1 only works with IPv4 loopback, but socket.gethostname()
    # should always work:
    input_socket.bind((socket.gethostname(), 0))
    random_port_number = input_socket.getsockname()[1]
    input_socket.listen(1)

    # Do something with input_socket, for example pass it to another thread.

    output_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    # close() should not strictly be necessary here, but since connect() could fail, it avoids leaking fds
    # in that case. "If a file descriptor is given, it is closed when the returned I/O object is closed".
    with output_socket:
        output_socket.connect((socket.gethostname(), random_port_number))

The user of input_socket (e.g. another thread) can then do:

with input_socket:
    while True:
        readables, _, _ = select.select([input_socket], [], [input_socket], 1.0)

        if len(readables) > 0:
            input_conn, addr = self.input_socket.accept()
            break

    with input_conn:
        while True:
            readables, _, errored = select.select([input_conn], [], [input_conn], 1.0)
            if len(errored) > 0:
                print("connection errored, stopping")
                break

            if len(readables) > 0:
                read_data = input_conn.recv(1024)
                if len(read_data) == 0:
                    print("connection closed, stopping")
                    break
                else:
                    print(f"read data: {read_data!r}")