Unicode exception in twisted

Question

On my production server I have periodically happening unicode error but not on my desktop. It appears in logs:

2011-03-17 13:14:53+0000 [GameProtocol,941,95.78.43.17] <unicode instance at 0x9e304a0 with str error:
     Traceback (most recent call last):
      File "/usr/local/lib/python2.6/dist-packages/twisted/python/reflect.py", line 546, in _safeFormat
        return formatter(o)
    UnicodeEncodeError: 'ascii' codec can't encode characters in position 21-26: ordinal not in range(128)
    >

It doesn't affect any logic in application but it's annoying in logs.

The server runs under Ubuntu 10.10 Server, Python 2.6.5, Twisted 10.2.0.

The desktop is Ubuntu 10.10 Desktop, Python 2.6.5, Twisted 10.2.0.

Locales are the same:

$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

What would be the problem?

Answer 1

]It's not safe to log unicode with the Twisted logging system. Here's the minimal example that produces the exception you're seeing:

Python 2.6.4 (r264:75706, Dec  7 2009, 18:43:55)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from twisted.python import log
>>> import sys
>>> log.startLogging(sys.stdout)
2011-03-17 11:03:47-0400 [-] Log opened.
>>> log.msg(u'\N{SNOWMAN}')
2011-03-17 11:03:53-0400 [-] <unicode instance at 140508177816384 with str error Traceback (most recent call last):
      File "/usr/lib/python2.6/dist-packages/twisted/python/reflect.py", line 560, in safe_str
        return str(o)
    UnicodeEncodeError: 'ascii' codec can't encode character u'\u2603' in position 0: ordinal not in range(128)
    >
>>> 

So you need to find out what unicode is being logged and stop doing that, probably by encoding it in what ever way you want your log files to be encoded.