UnicodeDecodeError exception in recent actions

[ctetruite]

Here is my situation.

I have a model with this verbose_name in meta data : u"Matériel d'occasion". The thing is this is a non ascii string. I do some stuff on an object of this model. Then, when I go the the admin page, I get this server error :

Caught UnicodeDecodeError while rendering: ('ascii', "Mat\xc3\xa9riel d'occasion", 3, 4, 'ordinal not in range(128)')

It occurs in the template django/contrib/admin/templates/admin/index.html at line 70 :

<span class="mini quiet">{% filter capfirst %}{% trans entry.content_type.name %}{% endfilter %}</span>

[ctetruite]

Stack trace

Traceback:
File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response
  100.                     response = callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py" in wrapper
  207.                 return self.admin_view(view, cacheable)(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/utils/decorators.py" in _wrapped_view
  76.                     response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  69.         response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py" in inner
  190.             return view(request, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  69.         response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py" in index
  396.             context_instance=context_instance
File "/usr/lib/python2.6/site-packages/django/shortcuts/__init__.py" in render_to_response
  20.     return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)
File "/usr/lib/python2.6/site-packages/django/template/loader.py" in render_to_string
  186.     return t.render(context_instance)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  173.             return self._render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in _render
  167.         return self.nodelist.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  796.                 bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py" in render_node
  72.             result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/loader_tags.py" in render
  125.         return compiled_parent._render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in _render
  167.         return self.nodelist.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  796.                 bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py" in render_node
  72.             result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/loader_tags.py" in render
  125.         return compiled_parent._render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in _render
  167.         return self.nodelist.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  796.                 bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py" in render_node
  72.             result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/loader_tags.py" in render
  62.             result = block.nodelist.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  796.                 bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py" in render_node
  72.             result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py" in render
  260.             return self.nodelist_false.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  796.                 bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py" in render_node
  72.             result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py" in render
  174.                 nodelist.append(node.render(context))
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py" in render
  258.             return self.nodelist_true.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  796.                 bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py" in render_node
  72.             result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py" in render
  82.         output = self.nodelist.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in render
  796.                 bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py" in render_node
  72.             result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/templatetags/i18n.py" in render
  45.         output = self.filter_expression.resolve(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in resolve
  553.                 obj = self.var.resolve(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py" in resolve
  701.             return _(value)
File "/usr/lib/python2.6/site-packages/django/utils/translation/__init__.py" in ugettext
  55.     return real_ugettext(message)
File "/usr/lib/python2.6/site-packages/django/utils/functional.py" in _curried
  55.         return _curried_func(*(args+moreargs), **dict(kwargs, **morekwargs))
File "/usr/lib/python2.6/site-packages/django/utils/translation/__init__.py" in delayed_loader
  36.     return getattr(trans, real_name)(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/utils/translation/trans_real.py" in ugettext
  276.     return do_translate(message, 'ugettext')
File "/usr/lib/python2.6/site-packages/django/utils/translation/trans_real.py" in do_translate
  262.         result = getattr(t, translation_function)(eol_message)
File "/usr/lib/python2.6/gettext.py" in ugettext
  404.             return unicode(message)

Exception Type: TemplateSyntaxError at /admin/
Exception Value: Caught UnicodeDecodeError while rendering: ('ascii', "Mat\xc3\xa9riel d'occasion", 3, 4, 'ordinal not in range(128)')

[Claude Paroz]

Add test for unicode model name

[Claude Paroz]

I've just added a test that specifically tests this. But on my system, it passes without your patch.

Why would entry.content_type.name not be an Unicode string in the first place?

The following snippet could indicate you if ContentType names are really all Unicode strings.

from django.contrib.contenttypes.models import ContentType
for c in ContentType.objects.all():
    print c.name.__class__, c.name

[ctetruite]

So I've learnt how to use the test suite. Here are my results. Adding your unit test, it also passes for me. It outputs that when displaying content types: <type 'unicode'> ¿Chapter?

Then, I changed the Chapter verbose name from '¿Chapter?' to u"Matériel d'occasion" with a french accent. And so on in the unit test. Executing tests give me this:

<type 'unicode'> Matériel d'occasion

But the test fails with this error:

AssertionError: Couldn't find '<span class="mini quiet">Matériel d'occasion</span>' in response

Did I configured models in the wrong way ?

[Claude Paroz]

That's a totally different error. Simple quotes are converted to &#39; in the HTML result. So your test case should test for:

self.assertContains(response, """<span class="mini quiet">Matériel d&#39;occasion</span>""")

[ctetruite]

Oops, sorry. No problem with the test suite: sqlite or mysql.

With my application in dev mode, using MySql (DEFAULT CHARACTER SET utf8 COLLATE utf8_bin), I tried accessing admin page. I've just printed content_type class value at the same place that in my patch. I get this:

Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
<type 'str'> Matériel d'occasion
<type 'str'> Matériel d'occasion
<type 'str'> Matériel d'occasion
[02/Dec/2010 21:15:48] "GET /admin/ HTTP/1.1" 500 394166

Whereas in the test suite it's <unicode>. In the database:

select * from django_content_type;
Matériel d'occasion | website      | occmaterial |
...

[Claude Paroz]

I suspect some database configuration problem. Did you try to get the class value of any other CharField on one of your own models?

[Claude Paroz]

Well, ramiro friendly pointed me to ​http://docs.djangoproject.com/en/1.2/ref/databases/#collation-settings
It is surely the cause of your problems (using utf8_bin collation).

[Ramiro Morales]

(In [14779]) Fixed grammar and tweaked notes about MySQL database/table collation interaction with text fields. Refs #14417.

[Ramiro Morales]

(In [14780]) [1.2.X] Fixed grammar and tweaked notes about MySQL database/table collation interaction with text fields. Refs #14417.

Backport of [14779] from trunk