Python decimal.Decimal id not the same

Question

I am having a problem with encoding to JSON in Python, specifically with decimal.Decimal values. I am using this to output JSON for a Google App Engine application.

To circumvent the exception from the default json module in Python telling me that it can't handle decimal.Decimal objects, I am using this encoder subclass:

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            return float(o)
        return super(DecimalEncoder, self).default(o)

On other applications, this does work. In this case it doesn't. After much frustration I found out that this gives weird results:

print id(decimal.Decimal)
print id(type(o))

One would expect the id's to be identical, because it makes sense for the same class object to reside only once in memory. Because the id's differ, isinstance() doesn't work.

Could it be that decimal.Decimal is already imported somewhere else, for instance in the App Engine and/or webapp2 packages?

The following modul reproduces the error on my system (OSx 10.10, Python 2.7.6, GAE SDK 1.9.20). Just create an GAE application and put this in main.py:

import webapp2, decimal, json, MySQLdb, sys

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        print id(decimal.Decimal)
        print id(type(o))
        if isinstance(o, decimal.Decimal):
            return float(o)
        return super(DecimalEncoder, self).default(o)

class MainHandler(webapp2.RequestHandler):
    def get(self):
        db = MySQLdb.connect(unix_socket='/var/mysql/mysql.sock', host='localhost', user='root', db='ssss', charset='utf8')
        cursor = db.cursor(MySQLdb.cursors.DictCursor)
        cursor.execute("SELECT id, price FROM product WHERE id = 1")
        record = cursor.fetchone()

        self.response.headers['Content-Type'] = 'application/json'
        self.response.write(json.dumps(
            record,
            cls=DecimalEncoder,
            indent=4,
            separators=(',', ': ')
        ))

app = webapp2.WSGIApplication([
    ('/', MainHandler)
], debug=True)

Database table:

CREATE TABLE `product` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `price` decimal(10,2) unsigned DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8;
INSERT INTO product VALUES(0, 5.00);

Answer 1

It appears that the decimal.Decimal class is patched somewhere in the Google App Engine SDK (or the module is reloaded), and this is done between the MySQL conversion library importing decimal and you importing the same.

Luckily, we can work around this by updating MySQL conversion table:

from MySQLdb.constants import FIELD_TYPE
from MySQLdb.converters import conversions
import decimal

conversions[FIELD_TYPE.DECIMAL] = conversions[FIELD_TYPE.NEWDECIMAL] = decimal.Decimal

That's it; the above code resets the MySQL class and your JSON encoder type check succeeds.

The alternative would be for you to look for the class MySQLdb is using:

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, MySQLdb.converters.conversions[MySQLdb.constants.FIELD_TYPE.DECIMAL]):
            return float(o)
        return super(DecimalEncoder, self).default(o)