[patch] psycopg2 backend doesn't properly save/load unicode strings

[crankycoder@…]

Django doesn't properly set the unicode encoding on the psycopg2 driver so when you save unicode strings into an attribute, you get raw strings back which you have to manually decode into proper unicode strings.

Here's what Django does now:

In [1]: from travel.models import *

In [2]: t = Trip.objects.get(pk=1)

In [3]: t.name
Out[3]: '\xc3\x90'

In [4]: t.name = u'\xd0'

In [5]: t.name
Out[5]: u'\xd0'

In [6]: t.save()

In [7]: t.name
Out[7]: u'\xd0'

In [8]: t = Trip.objects.get(pk=1)

In [9]: t.name
Out[9]: '\xc3\x90'

In [10]: t.name.decode("utf8")
Out[10]: u'\xd0'

Here's what it does after the patch:

In [1]: from travel.models import *

In [2]: t = Trip.objects.get(pk=1)

In [3]: t.name
Out[3]: u'Taste of Maple'

In [4]: t.name = u'\xd0'

In [5]: t.name
Out[5]: u'\xd0'

In [6]: t.save()

In [7]: t.name
Out[7]: u'\xd0'

In [8]: t = Trip.objects.get(pk=1)

In [9]: t.name
Out[9]: u'\xd0'

In [10]:

patch:

Index: base.py
===================================================================
--- base.py     (revision 3548)
+++ base.py     (working copy)
@@ -7,6 +7,9 @@
 from django.db.backends import util
 try:
     import psycopg2 as Database
+    #  Register unicode conversions
+    import psycopg2.extensions
+    psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
 except ImportError, e:
     from django.core.exceptions import ImproperlyConfigured
     raise ImproperlyConfigured, "Error loading psycopg2 module: %s" % e

[Adrian Holovaty]

(In [3565]) Fixed #2514 -- Improved psycopg2 backend to save/load Unicode strings correctly. Thanks, crankycoder@…

[Victor Ng <victor.ng@…>]

rolled back as of changeset [3675]

[Victor Ng <crankycoder@…>]

This change won't be put in until Django gets full unicode support. For now - either encode/decode to UTF8 when going to the database or call the psycopg2.extensions.register_type function in your own code.

See the ​django-dev thread for more details.

[Chris Wagner <cw264701@…>]

FYI, it looks like psycopg2.extensions.register_type(psycopg2.extensions.UNICODE) is now called in the postgres_psycopg2 backend -- probably because the Unicode branch has been merged into trunk.