Python String Formats with SQL Wildcards and LIKE

Question

I'm having a hard time getting some sql in python to correctly go through MySQLdb. It's pythons string formatting that is killing me.

My sql statement is using the LIKE keyword with wildcards. I've tried a number of different things in Python. The problem is once I get one of them working, there's a line of code in MySQLdb that burps on string format.

Attempt 1:

"SELECT tag.userId, count(user.id) as totalRows FROM user INNER JOIN tag ON user.id = tag.userId WHERE user.username LIKE '%%s%'" % (query)

This is a no go. I get value error:

ValueError: unsupported format character ''' (0x27) at index 128

Attempt 2:

"SELECT tag.userId, count(user.id) as totalRows FROM user INNER JOIN tag ON user.id = tag.userId WHERE user.username LIKE '\%%s\%'" % (query)

I get the same result from attempt 1.

Attempt 3:

like = "LIKE '%" + str(query) + "%'" totalq = "SELECT tag.userId, count(user.id) as totalRows FROM user INNER JOIN tag ON user.id = tag.userId WHERE user.username " + like

This correctly creates the totalq variable, but now when I go to run the query I get errors from MySQLdb:

File "build/bdist.macosx-10.6-universal/egg/MySQLdb/cursors.py", line 158, in execute query = query % db.literal(args) TypeError: not enough arguments for format string

Attempt 4:

like = "LIKE '\%" + str(query) + "\%'" totalq = "SELECT tag.userId, count(user.id) as totalRows FROM user INNER JOIN tag ON user.id = tag.userId WHERE user.username " + like

This is the same output as attempt 3.

This all seems really strange. How can I use wildcards in sql statements with python?

Answer 1

It's not about string formatting but the problem is how queries should be executed according to db operations requirements in Python (PEP 249)

try something like this:

sql = "SELECT column FROM table WHERE col1=%s AND col2=%s" 
params = (col1_value, col2_value)
cursor.execute(sql, params)

here are some examples for psycog2 where you have some explanations that should also be valid for mysql (mysqldb also follows PEP249 dba api guidance 2.0: here are examples for mysqldb)

Answer 2

Those queries all appear to be vulnerable to SQL injection attacks.

Try something like this instead:

curs.execute("""SELECT tag.userId, count(user.id) as totalRows 
                  FROM user 
            INNER JOIN tag ON user.id = tag.userId 
                 WHERE user.username LIKE %s""", ('%' + query + '%',))

Where there are two arguments being passed to execute().

Answer 3

To escape ampersands in Python string formatting expressions, double the ampersand:

'%%%s%%' % search_string

Edit: But I definitely agree with another answer. Direct string substitution in SQL queries is almost always a bad idea.

Answer 4

We could try escaping the percentage character by doubling them like this:

query_to_get_user_name = """ 
SELECT tag.userId, count(user.id) as totalRows FROM user INNER JOIN tag 
ON user.id = tag.userId 
WHERE user.username LIKE '%%%s%%' """ % (user_name,) 

cursor.execute(query_to_get_user_name)

Answer 5

I have a solution to your problem :

You can not use :

"SELECT tag.userId, count(user.id) as totalRows FROM user INNER JOIN tag ON user.id = tag.userId WHERE user.username LIKE '%%s%'" % (query)

you can change it with string template, such as :

import MySQLdb
import string # string module
.......
value = {'user':'your value'}
sql_template = string.Template("""
SELECT tag.userId, count(user.id) as totalRows FROM user INNER JOIN
tag ON user.id = tag.userId WHERE user.username LIKE '%$user%'
""")

sql = sql_template.substitute(value)

try:
    cursor.execute(sql)
    ...........
except:
    ...........
finally :
   db.close()