Postgres dump of only parts of tables for a dev snapshot

Question

On production our database is a few hundred gigabytes in size. For development and testing, we need to create snapshots of this database that are functionally equivalent, bu

Answer 1

http://jailer.sourceforge.net/ does this.

Answer 2

On your larger tables you can use the COPY command to pull out subsets...

COPY (SELECT * FROM mytable WHERE ...) TO '/tmp/myfile.tsv'

COPY mytable FROM 'myfile.tsv'

https://www.postgresql.org/docs/current/static/sql-copy.html

You should consider maintaining a set of development data rather than just pulling a subset of your production. In the case that you're writing unit tests, you could use the same data that is required for the tests, trying to hit all of the possible use cases.

Answer 3

I don't know about any software which already does this, but I can think of 3 alternative solutions. Unfortunately, they all require some custom coding.

1. Re-create all the tables in a separate schema, then copy into those tables only the subset of data you would like to dump, using INSERT INTO copy.tablename SELECT * FROM tablename WHERE ... and dump that.

2. Write your own script for dumping data as SQL statements. I have used this approach in the past and it only took something like 20-30 lines of PHP.

3. Modify pg_dump so it accepts a condition along with the -t switch when dumping a single table.