Unicode-errors with url-tags after changeset 8760

[Ville Säävuori]

I'm using url-templatetag across my projects. After Changeset r8760 many of (all?) those tags raise an UnicodeDecodeError.

Here is a snippet from the exception:

Exception Type: TemplateSyntaxError at /blogi/
Exception Value: Caught an exception while rendering: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

Original Traceback (most recent call last):
  File "/Library/Python/2.5/site-packages/django/template/debug.py", line 71, in render_node
    result = node.render(context)
  File "/Library/Python/2.5/site-packages/django/template/defaulttags.py", line 168, in render
    nodelist.append(node.render(context))
  File "/Library/Python/2.5/site-packages/django/template/defaulttags.py", line 168, in render
    nodelist.append(node.render(context))
  File "/Library/Python/2.5/site-packages/django/template/defaulttags.py", line 393, in render
    url = reverse(self.view_name, args=args, kwargs=kwargs)
  File "/Library/Python/2.5/site-packages/django/core/urlresolvers.py", line 250, in reverse
    *args, **kwargs)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

Tag used:

{% url aino.views.tag_detail tag=tag %}

Related line in URLConf:

url(r'^tagit/(?P<tag>\S+)/$', "aino.views.tag_detail", name="tag_detail"),

(Tags in above code may contain unicode characters.)

My test environment is on Mac OS X 10.5, python 2.5, every file is utf-8 encoded.

[Ville Säävuori]

This is a duplicate of #8725

[Karen Tracey]

Replying to Uninen:

This is a duplicate of #8725

Are you sure? The exception in #8725 is entirely different and doesn't appear to have anything to do with UnicodeDecode errors. This looks to me like a different problem, though yes it was introduced in the same changeset.

[Ville Säävuori]

On another look, it may be a different problem. Not sure, though. Can someone duplicate these UnicodeDecodeErrors?

[Carl Meyer]

I'm pretty sure this is a separate problem from #8725. I'm not able to duplicate it, but I'm not trying to use Unicode characters in a URL slug.

[Malcolm Tredinnick]

(In [8766]) Added a couple of extra tests to ensure url tag robustness. These are an
attempt to verify #8726.

Refs #8726.

[Malcolm Tredinnick]

We're going need to some more information to be able to repeat this, because the description doesn't have the critical piece. You can see in [8766] that there are now tests which do exactly what is described (there were previously tests that were very similar and using non-ASCII characters, but I added a couple that were even closer to this ticket) and they don't fail.

So if you could come up with a variation on those tests that does fail, that would be great. Failing that, a small example of what you are passing to the URL tag and what you URLConf looks like. If the critical URLConf is included inside another one, we need them all the way back to the top in case it is a variant on #8725.

If the problem's been fixed after [8763], also, please close this ticket.

[Ville Säävuori]

Thank you for specifying the problem. It seems that the core of this problem is that the argument for url-tag is *not* unicode but a regular string. Following test produces exactly the same error that I have on some of my project URLs:

(in /tests/regressiontests/templates/tests.py )

'url08': (u'{% url метка_оператора v %}', {'v': 'Ω'}, '/url_tag/%D0%AE%D0%BD%D0%B8%D0%BA%D0%BE%D0%B4/%CE%A9/'),

Why this is, I don't know (yet), because in my case the arguments are normal model fields or model methods which should in every case give unicode strings. But still, I think we should be liberal in what to accept for the url-tag (ie. force arguments to unicode instead of raising an exception), right? At least with previous revisions all of my url-tags worked OK.

[Ville Säävuori]

As a clarification, above test fails with r8759, too. I still think it *should* work, though.

Furthermore, I studied my projects more carefully and I found that the original problem originated when I called the templatetag with the instance of django-tagging object as the argument (like {% url viewname tag=tag %}). When I changed the argument to the name field (like {% url viewname tag=tag.name %}), it worked fine. The django-tagging model has an __unicode__ method (that returns self.name, not explicitly converting name to unicode). Hope this helps.

And again, these same urls worked fine before r8760.

[Malcolm Tredinnick]

Thanks for the test case. Now I can work with something. But if django-tagging is attempting to return something from a __unicode__ method that isn't unicode that's an outright bug in django-tagging. Unicode methods must return unicode strings. That isn't optional. Still, using UTF-8 strings in legal situations should work, so this is still a bug. But the situation you have it arising is indicative of a bug somewhere else.

[Malcolm Tredinnick]

(In [8777]) Fixed #8726 -- When doing reverse URL resolving, make sure we're consistently
dealing with unicode strings throughout by promoting up from UTF-8 as necessary.

[Jacob]

Milestone 1.0 deleted