How to copy a database with mysqldump and mysql in Python?

Question

I am writing a simple Python script to copy a MySQL database. I am attempting to copy the database based on the following SO questions and their answers: "Copy/duplicate database without using mysqldump", "python subprocess and mysqldump" and "Python subprocess, mysqldump and pipes". However, my script does not work for some reason I cannot see as the tables and the data do not appear in my new database.

I can see from my output that the mysqldump works correctly (I see a "Dump completed on..." in my output), so I think that something is wrong with my pipeline.

Here is my script:

#!/usr/bin/env python

import pymysql
from subprocess import Popen, PIPE, STDOUT

conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='', db='mydb')
cur = conn.cursor()

print("Attempting to create new database...")
try:
    cur.execute("CREATE DATABASE mydb2")
    print("Creating new database")
except Exception:
    print("Database already exists")
print()

# close connection just to be sure
cur.close()
conn.close()

print("Trying to copy old database to new database...")

args1 = ["mysqldump", "-h", "localhost", "-P", "3306", "-u", "root", "-p", "mydb"]
args2 = ["mysql", "-h", "localhost", "-P", "3306", "-u", "root", "-p", "mydb2"]

p1 = Popen(args1, stdout=PIPE, stderr=STDOUT)
p2 = Popen(args1, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)
output = p2.communicate()

print("output:")
print(output)
print()

As you can see I took the copy database pipeline from this answer. And at first I had the error mysqldump: Couldn't find table: "|" just as in that other question. So now I use two subprocess.Popen calls as suggested, which solved that error message.

The output variable shows that a mysqldump is performed, but I see nothing being mentioned about the mysql command.

I have tried to use p2.wait() and p1.wait() instead of p2.communicate() as suggested in one answer, but that just makes my Python script become unresponsive.

I have also tried the following:

output1 = p1.communicate()
output2 = p2.communicate()

But then both output1 and output2 show the same mysqldump output. So that was just a silly thing to do I guess..

I have also tried to use subprocess.call instead of subprocess.Popen, but that also makes my script become unresponsive.

Also including shell=True in either Popen or call also results in the script being just unresponsive.

However, it does work to type in the command in the command prompt (I use Windows 8.1) as follows:

mysqldump -h localhost -P 3306 -u root -p mydb | mysql -h localhost -P 3306 -u root -p mydb2

It copies my small test database in less than three seconds.

I wish I could also get it to work in Python.

Answer 1

I don't know the degree of pure Python you want to use for the copy, but you can just delegate the entire pipe operation to the shell.

subprocess.Popen('mysqldump -h localhost -P 3306 -u -root mydb | mysql -h localhost -P 3306 -u root mydb2', shell=True)

This should work the same way it works when you run it on the shell.

Answer 2

One problem that I saw is on this line:

p2 = Popen(args1, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)

It should read:

p2 = Popen(args2, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)

(args1 were being passed to the second proc, so that the program did two dumps and zero restores)

Answer 3

Here's how you could run mysqldump .. | mysql pipeline without the shell:

#!/usr/bin/env python
from subprocess import Popen, PIPE

mysql = Popen("mysql -h localhost -P 3306 -u root -p mydb2".split(),
              stdin=PIPE, stdout=PIPE)
mysqldump = Popen("mysqldump -h localhost -P 3306 -u root -p mydb".split(),
                  stdout=mysql.stdin)
mysql_stdout = mysql.communicate()[0]
mysqldump.wait()

See How do I use subprocess.Popen to connect multiple processes by pipes?

If you don't need to pass command-line parameters that require complex (possibly non-portable) escaping, capture the exit statuses, stdout then it is simpler to use the shell here.