Opened 6 years ago

Closed 5 years ago

Last modified 4 years ago

#12302 closed (fixed)

Exception with utf8 message leads to DjangoUnicodeDecodeError

Reported by: bjourne Owned by: nobody
Component: Database layer (models, ORM) Version: 1.1
Severity: Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:


The psycopg2 driver raises exceptions with i18n:ized messages. Django is not able to diplay them and will raise a DjangoUnicodeDecodeError. Example:

File "/var/lib/python-support/python2.6/django/views/", line 39, in technical_500_response
    html = reporter.get_traceback_html()
  File "/var/lib/python-support/python2.6/django/views/", line 97, in get_traceback_html
    'exception_value': smart_unicode(self.exc_value, errors='replace'),
  File "/var/lib/python-support/python2.6/django/utils/", line 35, in smart_unicode
    return force_unicode(s, encoding, strings_only, errors)
  File "/var/lib/python-support/python2.6/django/utils/", line 70, in force_unicode
    raise DjangoUnicodeDecodeError(s, *e.args)
DjangoUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128). You passed in DataError('FEL:  v\xc3\xa4rdet f\xc3\xb6r l\xc3\xa5ngt f\xc3\xb6r typen character varying(511)\n',) (<class 'psycopg2.DataError'>)

Change History (11)

comment:1 Changed 6 years ago by wdoekes

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Hmyes.. I've tested this on my debian/lenny install and can confirm this.

psycopg2 uses the client_encoding (as set through self.connection.set_client_encoding('UTF8') in django/db/backends/postgresql_psycopg2/ for its error messages.

Thus, to get this fixed, all DatabaseErrors raised by psycopg2 should have their .message.decode('utf-8')'d before they're exposed to the world.

comment:2 Changed 5 years ago by russellm

  • Resolution set to invalid
  • Status changed from new to closed

I can't reproduce this problem. I don't deny that the problem exists (since at least one person has been able to reproduce it), but the stack trace and comment doesn't give enough detail to explain how to reproduce the problem. If you can provide instructions on how to reproduce the problem, please reopen.

comment:3 Changed 5 years ago by kmtracey

  • Resolution invalid deleted
  • Status changed from closed to reopened

I think I know what this is. It's a recurrence of the problem fixed by r8588 that happens when running under Python 2.6. Prior to 2.6 (or with 2.5 at any rate), Exceptions did not have unicode methods, with 2.6 they do. The r8588 special-casing for Exceptions was made only in the path that applies to objects without unicode methods, thus it is no longer working under 2.6. (Having a unicode method under 2.6 is also why the original traceback is now completely absent in TemplateSyntaxErrors that wrap other exceptiosn -- the code to include the original traceback in the exception value is in an __str__ method, and 2.6 no longer calls that. But that's a different problem than this one, and there's also another ticket open for that.)

comment:4 Changed 5 years ago by kmtracey

  • milestone set to 1.2
  • Triage Stage changed from Unreviewed to Accepted

comment:5 Changed 5 years ago by russellm

  • Component changed from Uncategorized to Database layer (models, ORM)

comment:6 Changed 5 years ago by kmtracey

Not sure why you put this in the ORM component? It's a general problem when running under 2.6 with any exception that has non-ASCII data and does not provide a working unicode method. Strictly speaking I think the exceptions are broken, but Malcolm eventually agreed in #6353 that having Django's debug info code generate a 2nd exception because the 1st exception is broken is something that should be avoided. The fix probably goes into force_unicode, though I haven't had a chance to look at it closely yet.

comment:7 Changed 5 years ago by wdoekes

Hm. I cannot reproduce my reproduction. Must've been smoking crack at the time. force_unicode seems to do what it should correctly:

Set lc_messages and restart postgresql:

$ grep lc_messages /etc/postgresql/8.3/main/postgresql.conf 
lc_messages = 'sv_SE.UTF-8'                     # locale for system error message

Start up a django shell:

$ ./manage shell
Python 2.5.2 (r252:60911, Jan  4 2009, 21:59:32) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from django import VERSION
(1, 1, 1, 'final', 0)
>>> from django.db import connection
>>> c = connection.cursor()
>>> c.execute('SELECT 123 FROM;')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/django11/django/db/backends/", line 19, in execute
    return self.cursor.execute(sql, params)
ProgrammingError: FEL:  syntaxfel vid eller nära ";"

>>> try: c.execute('SELECT 123 FROM;')
... except Exception, e: pass
>>> unicode(e)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 27: ordinal not in range(128)
>>> from django.utils.encoding import force_unicode
>>> force_unicode(e)
u'FEL:  syntaxfel vid eller n\xe4ra ";"\nLINE 1: SELECT 123 FROM;\n                       ^\n'

I do not deny that there is the problem you've analyzed. But I retract my reproduction.

comment:8 Changed 5 years ago by kmtracey

The shell session you show is using Python 2.5. The original traceback is using Python 2.6. (I can't tell what you were using in the first comment where you could reproduce it.) The problem only occurs under 2.6 and higher; it is due to a change in Python that the fix made in r8588 is not kicking in. Specifically, exceptions in 2.6+ have unicode methods, where they did not in 2.5, and the r8588 fix was made in a path that deals only with objects that do not have unicode methods.

comment:9 Changed 5 years ago by kmtracey

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [12621]) Fixed #12302: Modified force_unicode to avoid raising unicode errors when
handed exceptions with non-ASCII bytestring data and no working unicode method
under Python 2.6 and higher.

comment:10 Changed 5 years ago by kmtracey

(In [12623]) [1.1.X] Fixed #12302: Modified force_unicode to avoid raising unicode errors when
handed exceptions with non-ASCII bytestring data and no working unicode method
under Python 2.6 and higher.

r12621 and r12622 from trunk.

comment:11 Changed 4 years ago by jacob

  • milestone 1.2 deleted

Milestone 1.2 deleted

Note: See TracTickets for help on using tickets.
Back to Top