Opened 8 years ago

Closed 8 years ago

#4664 closed (fixed)

[unicode] UnicodeDecodeError with on postgresql 7.4 using psycopg1 backend on python 2.3

Reported by: Øyvind Saltvik <oyvind@…> Owned by: mtredinnick
Component: Database layer (models, ORM) Version: unicode
Severity: Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:


Traceback (most recent call last):
File "/var/www/vhosts/" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/var/www/vhosts/" in contact
  50. return object_list(request, **kwargs)
File "/var/www/vhosts/" in object_list
  75. }, context_processors)
File "/var/www/vhosts/" in __init__
  100. self.update(processor(request))
File "/var/www/vhosts/" in meny
  9. menyliste = list(Side.objects.filter(i_meny=True)) + list(Element.objects.all())
File "/var/www/vhosts/" in __iter__
  109. return iter(self._get_data())
File "/var/www/vhosts/" in _get_data
  471. self._result_cache = list(self.iterator())
File "/var/www/vhosts/" in iterator
  188. rows = cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)
File "/var/www/vhosts/" in typecast_string
  273. return smart_unicode(s, client_encoding)
File "/var/www/vhosts/" in smart_unicode
  25. return force_unicode(s, encoding, strings_only, errors)
File "/var/www/vhosts/" in force_unicode
  42. s = unicode(s, encoding, errors)

  UnicodeDecodeError at /kontakt/
  'ascii' codec can't decode byte 0xc3 in position 134: ordinal not in range(128)

the variable encoding is ascii on line 42

Change History (5)

comment:1 Changed 8 years ago by Øyvind Saltvik <oyvind@…>

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Version changed from SVN to unicode

comment:2 Changed 8 years ago by Øyvind Saltvik <oyvind@…>

Switching to psycopg2 solved the problem.

comment:3 Changed 8 years ago by mtredinnick

  • Owner changed from adrian to mtredinnick
  • Summary changed from unicode: UnicodeDecodeError with on postgresql 7.4 using psycopg1 backend on python 2.3 to [unicode] UnicodeDecodeError with on postgresql 7.4 using psycopg1 backend on python 2.3
  • Triage Stage changed from Unreviewed to Accepted

So for some reason your default client encoding is ASCII (possibly configured to be non-international friendly in postgresql.conf). You, or any other user, may not be in control of that, so we have to work around. It's going to be annoying to have to try and change this value, though. We don't know what is available on the server-side and things will go badly wrong (Django will crash, most likely) if we get it wrong.

I don't have PostgreSQL 7.4 on any machine at the moment. Can you add a line just before line 97 (at the same indentation level that as line 97) that says

cursor.execute("SET CLIENT_ENCODING to 'UNICODE'")

So you will have a "set client encoding" followed by "show client encoding". I want to see if the "set" takes effect. If so, we can try to set the encoding to UTF-8 if the client reports ASCII.

(I may have the "UNICODE" string wrong. It isn't listed as an explicit option in table 20.2, here, but it looks like it should be supported. Try a few alternative strings if that doesn't work.)

comment:4 Changed 8 years ago by Øyvind Saltvik <oyvind@…>

that solved it

comment:5 Changed 8 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [5535]) Fixed #4664 -- Forced the client character set encoding to UTF-8 for PostgreSQL
(via the psycopg backend). The previous version was causing problems on some
setups, particularly PostgreSQL 7.x. Current code should work with 7.x and 8.x,
no matter what the default client encoding is.

Note: See TracTickets for help on using tickets.
Back to Top