python inserting and retrieving binary data into mysql

Question

I\'m using the MySQLdb package for interacting with MySQL. I\'m having trouble getting the proper type conversions.

I am using a 16-byte binary uuid as a primary key

Answer 1

To supplement existing answers, there's also an issue with the following warning when dealing with binary strings in queries:

Warning: (1300, "Invalid utf8 character string: 'ABCDEF'") 

It is reproduced by the following:

cursor.execute('''
    CREATE TABLE `table`(
        bin_field` BINARY(16) NOT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
''')

bin_value = uuid.uuid4().bytes
cursor.execute('INSERT INTO `table`(bin_field) VALUES(%s)', (bin_value,))

Whenever MySQL sees that a string literal in a query isn't valid against current character_set_connection it will emit the warning. There are several solutions to it:

1. Explicitly set _binary charset literal

    INSERT INTO `table`(bin_field) VALUES(_binary %s)
   

2. Manually construct queries with hexadecimal literals

    INSERT INTO `table`(bin_field) VALUES(x'abcdef')
   

3. Change connection charset if you're only working with binary strings

For more details see MySQL Bug 79317.

Update

As @charlax pointed out, there's binary_prefix flag which can be passed to the connection's initialiser to automatically prepend _binary prefix when interpolating arguments. It's supported by recent versions of both, mysql-client and pymysql.

Answer 2

One tip: you should be able to call uuid.uuid4().bytes to get the raw bytes. As for timestamps, if you want to perform time/date manipulation in SQL it's often easier to deal with real TIMESTAMP types.

I created a test table to try to reproduce what you're seeing:

CREATE TABLE xyz (
    added_id INT AUTO_INCREMENT NOT NULL,
    id BINARY(16) NOT NULL,
    PRIMARY KEY (added_id),
    UNIQUE (id)
) DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ENGINE=InnoDB;

My script is able to insert and query for the rows using the binary field as a key without problem. Perhaps you are incorrectly fetching / iterating over the results returned by the cursor?

import binascii
import MySQLdb
import uuid

conn = MySQLdb.connect(host='localhost')

key = uuid.uuid4()
print 'inserting', repr(key.bytes)
r = conn.cursor()
r.execute('INSERT INTO xyz (id) VALUES (%s)', key.bytes)
conn.commit()

print 'selecting', repr(key.bytes)
r.execute('SELECT added_id, id FROM xyz WHERE id = %s', key.bytes)
for row in r.fetchall():
    print row[0], binascii.b2a_hex(row[1])

Output:

% python qu.py    
inserting '\x96\xc5\xa4\xc3Z+L\xf0\x86\x1e\x05\xebt\xf7\\\xd5'
selecting '\x96\xc5\xa4\xc3Z+L\xf0\x86\x1e\x05\xebt\xf7\\\xd5'
1 96c5a4c35a2b4cf0861e05eb74f75cd5
% python qu.py
inserting '\xac\xc9,jn\xb2O@\xbb\xa27h\xcd<B\xda'
selecting '\xac\xc9,jn\xb2O@\xbb\xa27h\xcd<B\xda'
2 acc92c6a6eb24f40bba23768cd3c42da