piping postgres COPY in python with psycopg2

Question

I\'m writing a script to do a copy of some data between two machines on the same network using psycopg2. I\'m replacing some old, ugly bash that does the copy with

Answer 1

You could use a deque that you've subclassed to support reading and writing:

from collections import deque
from Exceptions import IndexError

class DequeBuffer(deque):
    def write(self, data):
        self.append(data)
    def read(self):
        try:
            return self.popleft()
        except IndexError:
            return ''

buf = DequeBuffer()

If the reader is much faster than the writer, and the table is large, the deque will still get big, but it will be smaller than storing the whole thing.

Also, I don't know for sure return '' when the deque is empty is safe, rather than retrying until it's not empty, but I'd guess it is. Let me know if it works.

Remember to del buf when you're sure the copy is done, especially if the script isn't just exiting at that point.