Python MysqlDB using cursor.rowcount with SSDictCursor returning wrong count

Question

I have the following code

cur = db.cursor(cursors.SSDictCursor)
cur.execute("SELECT * FROM large_table")
result_count = cur.rowcount
print result_count

This prints the number 18446744073709551615 which is obviously wrong. If I remove the cursors.SSDictCursor the correct number is shown. Can anyone tell me how I can get the number of records returned while keeping the SSDictCursor?

Answer 1

To get the number of records returned by SSDictCursor or SSCursor, your only options are:

1. Fetch the entire result and count it using len(), which defeats the purpose of using SSDictCursor or SSCursor in the first place;

2. Count the rows yourself as you iterate through them, which means you won't know the count until hit the end (not likely to be practical); or,

3. Run an additional, separate COUNT(*) query.

I highly recommend the third option. It's extremely fast if all you're doing is SELECT COUNT(*) FROM table;. It would be slower for some more complex query, but with proper indexing it should still be quick enough for most purposes.

---

As an aside, the return value you're seeing is sort of correct; at least, as far as the MySQL C API is concerned.

Per the Python DB API defined in PEP 249, the rowcount attribute is -1 if the rowcount of the last operation cannot be determined by the interface. @glglgl explained why the rowcount can't be determined in their answer:

Internally, SSDictCursor uses mysql_use_result() which allows the server to start transferring the data before the acquiring is complete.

In other words, the server doesn't know how many rows it's ultimately going to fetch. When you execute a query, MySQLdb stores the return value of mysql_affected_rows() in the cursor's rowcount attribute. Because the count is indeterminate, this function returns -1 as an unsigned long long integer (my_ulonglong), a numeric type that's available in the ctypes module of the standard library:

>>> from ctypes import c_ulonglong
>>> n = c_ulonglong(-1)
>>> n.value
18446744073709551615L

A quick-and-dirty alternative to ctypes, when you know you'll always be dealing with a 64-bit unsigned integer, is:

>>> -1 & 0xFFFFFFFFFFFFFFFF
18446744073709551615L

It would be great if MySQLdb checked for this return value and gave you the signed integer you expect to see, but unfortunately it doesn't.

Answer 2

With a SSDictCursor, this value can only be read resp. determined when the cursor is used up.

Internally, SSDictCursor uses mysql_use_result() which allows the server to start transferring the data before the acquiring is complete.