#12302 closed (fixed)
Exception with utf8 message leads to DjangoUnicodeDecodeError
Reported by: | bjourne | Owned by: | nobody |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | 1.1 |
Severity: | Keywords: | ||
Cc: | Triage Stage: | Accepted | |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
The psycopg2 driver raises exceptions with i18n:ized messages. Django is not able to diplay them and will raise a DjangoUnicodeDecodeError. Example:
File "/var/lib/python-support/python2.6/django/views/debug.py", line 39, in technical_500_response html = reporter.get_traceback_html() File "/var/lib/python-support/python2.6/django/views/debug.py", line 97, in get_traceback_html 'exception_value': smart_unicode(self.exc_value, errors='replace'), File "/var/lib/python-support/python2.6/django/utils/encoding.py", line 35, in smart_unicode return force_unicode(s, encoding, strings_only, errors) File "/var/lib/python-support/python2.6/django/utils/encoding.py", line 70, in force_unicode raise DjangoUnicodeDecodeError(s, *e.args) DjangoUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128). You passed in DataError('FEL: v\xc3\xa4rdet f\xc3\xb6r l\xc3\xa5ngt f\xc3\xb6r typen character varying(511)\n',) (<class 'psycopg2.DataError'>)
Change History (12)
comment:1 by , 15 years ago
comment:2 by , 15 years ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
I can't reproduce this problem. I don't deny that the problem exists (since at least one person has been able to reproduce it), but the stack trace and comment doesn't give enough detail to explain how to reproduce the problem. If you can provide instructions on how to reproduce the problem, please reopen.
comment:3 by , 15 years ago
Resolution: | invalid |
---|---|
Status: | closed → reopened |
I think I know what this is. It's a recurrence of the problem fixed by r8588 that happens when running under Python 2.6. Prior to 2.6 (or with 2.5 at any rate), Exceptions did not have unicode methods, with 2.6 they do. The r8588 special-casing for Exceptions was made only in the path that applies to objects without unicode methods, thus it is no longer working under 2.6. (Having a unicode method under 2.6 is also why the original traceback is now completely absent in TemplateSyntaxErrors that wrap other exceptiosn -- the code to include the original traceback in the exception value is in an __str__
method, and 2.6 no longer calls that. But that's a different problem than this one, and there's also another ticket open for that.)
comment:4 by , 15 years ago
milestone: | → 1.2 |
---|---|
Triage Stage: | Unreviewed → Accepted |
comment:5 by , 15 years ago
Component: | Uncategorized → Database layer (models, ORM) |
---|
comment:6 by , 15 years ago
Not sure why you put this in the ORM component? It's a general problem when running under 2.6 with any exception that has non-ASCII data and does not provide a working unicode method. Strictly speaking I think the exceptions are broken, but Malcolm eventually agreed in #6353 that having Django's debug info code generate a 2nd exception because the 1st exception is broken is something that should be avoided. The fix probably goes into force_unicode, though I haven't had a chance to look at it closely yet.
comment:7 by , 15 years ago
Hm. I cannot reproduce my reproduction. Must've been smoking crack at the time. force_unicode seems to do what it should correctly:
Set lc_messages and restart postgresql:
$ grep lc_messages /etc/postgresql/8.3/main/postgresql.conf lc_messages = 'sv_SE.UTF-8' # locale for system error message
Start up a django shell:
$ ./manage shell Python 2.5.2 (r252:60911, Jan 4 2009, 21:59:32) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from django import VERSION >>> VERSION (1, 1, 1, 'final', 0) >>> from django.db import connection >>> c = connection.cursor() >>> c.execute('SELECT 123 FROM;') Traceback (most recent call last): File "<console>", line 1, in <module> File "/opt/django11/django/db/backends/util.py", line 19, in execute return self.cursor.execute(sql, params) ProgrammingError: FEL: syntaxfel vid eller nära ";" LINE 1: SELECT 123 FROM; ^ >>> try: c.execute('SELECT 123 FROM;') ... except Exception, e: pass ... >>> unicode(e) Traceback (most recent call last): File "<console>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 27: ordinal not in range(128) >>> from django.utils.encoding import force_unicode >>> force_unicode(e) u'FEL: syntaxfel vid eller n\xe4ra ";"\nLINE 1: SELECT 123 FROM;\n ^\n'
I do not deny that there is the problem you've analyzed. But I retract my reproduction.
comment:8 by , 15 years ago
The shell session you show is using Python 2.5. The original traceback is using Python 2.6. (I can't tell what you were using in the first comment where you could reproduce it.) The problem only occurs under 2.6 and higher; it is due to a change in Python that the fix made in r8588 is not kicking in. Specifically, exceptions in 2.6+ have unicode methods, where they did not in 2.5, and the r8588 fix was made in a path that deals only with objects that do not have unicode methods.
comment:9 by , 15 years ago
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
Hmyes.. I've tested this on my debian/lenny install and can confirm this.
psycopg2 uses the client_encoding (as set through self.connection.set_client_encoding('UTF8') in django/db/backends/postgresql_psycopg2/base.py) for its error messages.
Thus, to get this fixed, all DatabaseErrors raised by psycopg2 should have their .message.decode('utf-8')'d before they're exposed to the world.