UnicodeDecodeError in sql logger

[zw0rk]

Trying to filter a queryset with unicode values causes UnicodeDecodeError in sql logger.

>>> User.objects.filter(last_name=u'Z')
[]
>>> User.objects.filter(last_name=u'й')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 74, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 89, in __len__
    self._result_cache.extend(self._iter)
  File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 294, in iterator
    for row in compiler.results_iter():
  File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 764, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 819, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python2.7/site-packages/django/db/backends/util.py", line 51, in execute
    logger.debug('(%.3f) %s; args=%s' % (duration, sql, params),
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 850: ordinal not in range(128)

This happens even right after syncdb, when django tries to create admin_permission objects and models have unicode verbose_name's.

No custom loggers were installed in settings.py.

[Claude Paroz]

It seems that at least PostgreSQL and MySQL are returning their last executed query as an 'utf-8' encoded byte string. Needs confirmation by ORM guys, and check with Oracle.

[Claude Paroz]

Decode returned value from last executed query

[Simon Charette]

I've got bit by that using master and trying to load fixtures with non-ascii character.

[Claude Paroz <claude@…>]

In [e9ef9776d186a3379cdbf47a73b14e89b74d0926]:

Fixed #18461 -- Ensured that last_executed_query returns Unicode

Thanks Anssi Kääriäinen for the review.

[Anssi Kääriäinen <akaariai@…>]

In [86c20e39eb9ea0bbd25889403d16d857748aff9b]:

Fixed connection.queries encoding handling on Oracle

In addition, removed a possibly problematic .filter() call from
backends.test_query_encoding test. It is possible the .filter could
cause collation problems on MySQL, and as it wasn't absolutely needed
for the test it seemed better to get rid of the call.

Refs #18461.

[Anssi Kääriäinen]

For future reference: the problem with the test was that it used .filter(name='й'). This caused problems on MySQL when TEST_CHARSET = 'UTF8' is not in use. I didn't have that, and didn't spot that this is required (documented ​here). So, the original test was correct. I don't see a need to change the test back, as the removed .filter() doesn't make the test incorrect either. Of course, I am not objecting changing the test back if that is wanted...

[err]

This is doesn't work when you have connection not in utf8 (for example in mysql OPTIONS' : {"charset": "cp1251"})

'utf8' codec can't decode byte 0xcd in position 401: invalid continuation byte

[err]

<         encoding = cursor.connection.character_set_name()
<         return cursor._last_executed.decode(encoding)
---
>         return cursor._last_executed.decode('utf-8')

I guess that it will help

[Claude Paroz]

Please reopen a new ticket with your specific issue.