Django <-> SQL Server 2005, text encoding problem

Question

I'm trying to store Django data on MS SQL Server 2005 using:

http://code.google.com/p/django-pyodbc/ (pyodbc + FreeTDS)

As long as I'm storing string consist of ASCII characters everything is ok. When I'm using unicode (ex. '\xc5\x82'), django throws ProgrammingError on:

ProgrammingError at /admin/cli/punktrejestracji/add/
('42000', '[42000] [FreeTDS][SQL Server]The incoming tabular data stream (TDS) protocol stream is incorrect. The stream ended unexpectedly. (4002) (SQLExecDirectW)')

last element of trace is:

params  ('\xc5\x82',)
self    <django.db.backends.sql_server.pyodbc.base.CursorWrapper object at 0x92ef8ec>
sql 'SELECT (1) AS [a] FROM [cli_punktrejestracji] WHERE [cli_punktrejestracji].[adres] = ? '

BTW http://code.google.com/p/django-mssql/ doesn't seems to work under Linux, django-mssql needs pythoncom library. Am I right?

Answer 1

We use Django with SQL Server 2005. We found the same problem you did.

What ODBC driver are you using? FreeTDS?

We tried finding a good ODBC driver for linux/unix to use that would not throw the error above (and others) when unicode would come into play - and failed miserably. None of the drivers we tested - at least three, I can dig the names up if you would like - had any success in dealing with unicode strings via django-pyodbc.

What we ended up doing, sad as it might sound, was to decide to run Django on a Windows server (Apache + mod_wsgi) and use the Microsoft's SQL Native ODBC driver.

It works just fine - unicode wise - when we do that.

Answer 2

OK, the solution was found. In file freetds.conf there is

client charset = UTF-8

and it works exactly like it should.

Answer 3

in addition to the accepted response, it is possible to fix this error dirrectly in the settings.py :

DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        'NAME': 'MyTableName',
        'HOST': r'server.lan\server_instance_name',
        'USER': 'sa',
        'PASSWORD': 'P@SsW0Rd',
        'OPTIONS': {
            'host_is_server': True,
            "extra_params":"TDS_Version=8.0;ClientCharset=UTF-8",
            "autocommit": True,
            "driver_needs_utf8":True,
        },

     }
}

take a look at the extra_params

this don't rely on the global freetds.conf file, so it is better