piping postgres COPY in python with psycopg2

Question

I\'m writing a script to do a copy of some data between two machines on the same network using psycopg2. I\'m replacing some old, ugly bash that does the copy with

Answer 1

You will have to put one of your calls in a separate thread. I just realized you can use os.pipe(), which makes the rest quite straightforward:

#!/usr/bin/python
import psycopg2
import os
import threading

fromdb = psycopg2.connect("dbname=from_db")
todb = psycopg2.connect("dbname=to_db")

r_fd, w_fd = os.pipe()

def copy_from():
    cur = todb.cursor()
    cur.copy_from(os.fdopen(r_fd), 'table')
    cur.close()
    todb.commit()

to_thread = threading.Thread(target=copy_from)
to_thread.start()

cur = fromdb.cursor()
write_f = os.fdopen(w_fd, 'w')
cur.copy_to(write_f, 'table')
write_f.close()   # or deadlock...

to_thread.join()