What's the best way to copy a subset of a table's rows from one database to another in Postgres?

Question

I\'ve got a production DB with, say, ten million rows. I\'d like to extract the 10,000 or so rows from the past hour off of production and copy them to my local box. How do I do

Answer 1

With the constraint you added (not being superuser), I do not find a pure-SQL solution. But doing it in your favorite language is quite simple. You open a connection to the "old" database, another one to the new database, you SELECT in one and INSERT in the other. Here is a tested-and-working solution in Python.

 #!/usr/bin/python

""" 

Copy a *part* of a database to another one. See


With PostgreSQL, the only pure-SQL solution is to use COPY, which is
not available to the ordinary user.

Stephane Bortzmeyer 

"""

table_name = "Tests"
# List here the columns you want to copy. Yes, "*" would be simpler
# but also more brittle.
names = ["id", "uuid", "date", "domain", "broken", "spf"]
constraint = "date > '2009-01-01'"

import psycopg2

old_db = psycopg2.connect("dbname=dnswitness-spf")
new_db = psycopg2.connect("dbname=essais")
old_cursor = old_db.cursor()
old_cursor.execute("""SET TRANSACTION READ ONLY""") # Security
new_cursor = new_db.cursor()
old_cursor.execute("""SELECT %s FROM %s WHERE %s """ % \
                       (",".join(names), table_name, constraint))
print "%i rows retrieved" % old_cursor.rowcount
new_cursor.execute("""BEGIN""")
placeholders = []
namesandvalues = {}
for name in names:
    placeholders.append("%%(%s)s" % name)
for row in old_cursor.fetchall():
    i = 0
    for name in names:
        namesandvalues[name] = row[i]
        i = i + 1
    command = "INSERT INTO %s (%s) VALUES (%s)" % \
              (table_name, ",".join(names), ",".join(placeholders))
    new_cursor.execute(command, namesandvalues)
new_cursor.execute("""COMMIT""")
old_cursor.close()
new_cursor.close()
old_db.close()
new_db.close()