UnicodeDecodeError attempting to render VariableDoesNotExist as a string

[Karen Tracey <kmtracey@…>]

Given a simple template (simple.tmpl):

<table>
{% for item in item_list %}
<tr class="{% cycle 'even' 'odd' rowvar as rowcolors %}">
<td>{{ item }}</td>
</tr>
{% endfor %}
</table>

Attempting to render the template using a context that does not include the variable rowvar produces a TemplateSyntaxError resulting from raising of a VariableDoesNotExist, as expected:

>>> from django.template import Context, loader
>>> t = loader.get_template('simple.tmpl')
>>> t.render(Context({'item_list': [1,2,3]}))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 174, in render
    return self.nodelist.render(context)
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 792, in render
    bits.append(self.render_node(node, context))
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 820, in render_node
    raise wrapped
TemplateSyntaxError: Caught an exception while rendering: Failed lookup for key [rowvar] in [{'forloop': {'parentloop': {}, 'last': True, 'counter': 3, 'revcounter0': 0, 'revcounter': 1, 'counter0': 2, 'first': False}, u'item': 3, u'rowcolors': u'odd'}, {'item_list': [1, 2, 3]}]

Original Traceback (most recent call last):
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 810, in render_node
    result = node.render(context)
  File "/homedir/django/newforms-admin/django/template/defaulttags.py", line 135, in render
    nodelist.append(node.render(context))
  File "/homedir/django/newforms-admin/django/template/defaulttags.py", line 33, in render
    value = Variable(value).resolve(context)
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 704, in resolve
    return self._resolve_lookup(context)
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 754, in _resolve_lookup
    raise VariableDoesNotExist("Failed lookup for key [%s] in %r", (bit, current)) # missing attribute
VariableDoesNotExist: Failed lookup for key [rowvar] in [{'forloop': {'parentloop': {}, 'last': True, 'counter': 3, 'revcounter0': 0, 'revcounter': 1, 'counter0': 2, 'first': False}, u'item': 3, u'rowcolors': u'odd'}, {'item_list': [1, 2, 3]}]

If we now make the same error, but also include in the context an object whose representation includes non-ASCII data, the TemplateSyntaxError error is masked by a UnicodeDecodeError when the VariableDoesNotExist object attempts to return a string that include a representation of the context:

>>> from crossword.models import Authors
>>> a = Authors.objects.get(pk=535)
>>> a
<Authors: Sue de Nîmes>
>>> t.render(Context({'item_list': [1,2,3], 'xtra': a}))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 174, in render
    return self.nodelist.render(context)
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 792, in render
    bits.append(self.render_node(node, context))
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 817, in render_node
    wrapped = TemplateSyntaxError('Caught an exception while rendering: %s' % e)
  File "/homedir/django/newforms-admin/django/template/__init__.py", line 130, in __str__
    return self.msg % self.params
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 213: ordinal not in range(128)
>>>

My Authors model contains the following unicode method:

    def __unicode__(self):
        return u'%s' % self.Author

and the Author field is a CharField:

    Author = models.CharField(unique=True, maxlength=50)

I'm not sure what the correct fix is, but I'd be willing to create a patch if someone would point me in the right direction for how to fix it. One possibility would be to stop including the string representation of the context in the error message. I don't really see how it is useful...the key being sought isn't in there, what's the value of showing all the stuff (and it can be an awful lot of stuff, repeated three times on the debug page) that is in there?

[Mikkel Høgh]

I'll second that.

I think Karen is right - we don't need the string representation. Django's debug pages already have all the information we need and the string representation of the original object just makes it harder to read...

[Thomas Güttler]

The code uses newforms-admin and is several months old. Newforms-admin is not ready for production. I close it.

[Karen Tracey <kmtracey@…>]

While it is true the test was run with newforms-admin branch, that had no bearing on the problem; the code that is causing the problem isn't something introduced by newforms-admin. And even if it was a newforms-admin problem, the ticket shouldn't be closed. newforms-admin needs to be eventually merged to trunk so problems with it need to be fixed.

Problem still exists in today's trunk (revision 7169):

>>> from django.template import Context, loader
>>> t = loader.get_template('simple.tmpl')
>>> from crossword.models import Authors
>>> a = Authors.objects.get(pk=535)
>>> a
<Authors: Sue de Nîmes>
>>> t.render(Context({'item_list': [1,2,3], 'xtra': a}))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/homedir/django/trunk/django/template/__init__.py", line 173, in render
    return self.nodelist.render(context)
  File "/homedir/django/trunk/django/template/__init__.py", line 748, in render
    bits.append(self.render_node(node, context))
  File "/homedir/django/trunk/django/template/debug.py", line 78, in render_node
    wrapped = TemplateSyntaxError('Caught an exception while rendering: %s' % e)
  File "/homedir/django/trunk/django/template/__init__.py", line 132, in __str__
    return self.msg % self.params
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 213: ordinal not in range(128)

[Thomas Güttler]

Please test this patch.

[Karen Tracey <kmtracey@…>]

With the patch, the VariableDoesNotExist exception still gets suppressed, only now by a DjangoUnicodeDecodeError, which at least gives a hint that the first problem encountered was the VariableDoesNotExist:

>>> from django.template import Context, loader
>>> t = loader.get_template('simple.tmpl')
>>> from crossword.models import Authors
>>> a = Authors.objects.get(pk=535)
>>> a
<Authors: Sue de Nîmes>
>>> t.render(Context({'item_list': [1,2,3], 'xtra': a}))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/homedir/django/trunk/django/template/__init__.py", line 173, in render
    return self.nodelist.render(context)
  File "/homedir/django/trunk/django/template/__init__.py", line 748, in render
    bits.append(self.render_node(node, context))
  File "/homedir/django/trunk/django/template/debug.py", line 78, in render_node
    wrapped = TemplateSyntaxError(u'Caught an exception while rendering: %s' % force_unicode(e, errors='replace'))
  File "/homedir/django/trunk/django/utils/encoding.py", line 60, in force_unicode
    raise DjangoUnicodeDecodeError(s, *e.args)
DjangoUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 213: ordinal not in range(128). You passed in VariableDoesNotExist() (<class 'django.template.VariableDoesNotExist'>)

I had tried similar fixes when I first ran into this and for some reason couldn't find one that worked. However I took another look today and now I find that if I liberally sprinkle force_unicode down in VariableDoesNotExist that fixes the UnicodeDecode error and the real VariableDoesNotExist is preserved:

>>> from django.template import Context, loader
>>> t = loader.get_template('simple.tmpl')
>>> from crossword.models import Authors
>>> a = Authors.objects.get(pk=535)
>>> t.render(Context({'item_list': [1,2,3], 'xtra': a}))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/homedir/django/trunk/django/template/__init__.py", line 173, in render
    return self.nodelist.render(context)
  File "/homedir/django/trunk/django/template/__init__.py", line 748, in render
    bits.append(self.render_node(node, context))
  File "/homedir/django/trunk/django/template/debug.py", line 81, in render_node
    raise wrapped
TemplateSyntaxError: Caught an exception while rendering: Failed lookup for key [rowvar] in u"[{'forloop': {'revcounter0': 0, 'last': True, 'counter': 3, 'parentloop': {}, 'revcounter': 1, 'counter0': 2, 'first': False}, u'item': 3, u'rowcolors': u'odd'}, {'item_list': [1, 2, 3], 'xtra': <Authors: Sue de N\xeemes>}]"

Original Traceback (most recent call last):
  File "/homedir/django/trunk/django/template/debug.py", line 71, in render_node
    result = node.render(context)
  File "/homedir/django/trunk/django/template/defaulttags.py", line 149, in render
    nodelist.append(node.render(context))
  File "/homedir/django/trunk/django/template/defaulttags.py", line 47, in render
    value = Variable(value).resolve(context)
  File "/homedir/django/trunk/django/template/__init__.py", line 656, in resolve
    value = self._resolve_lookup(context)
  File "/homedir/django/trunk/django/template/__init__.py", line 709, in _resolve_lookup
    raise VariableDoesNotExist("Failed lookup for key [%s] in %r", (bit, current)) # missing attribute
VariableDoesNotExist

I'm not sure all of the force_unicodes I put in are necessary, and I don't know if there could ever be anything in a context that would not support having force_unicode applied to it, so someone with more of a clue in this area of code should verify the fix is OK. I also still don't think it is necessary to print out the whole context in this situation, but if it is desired to keep that behavior then I've got a possible fix for the problem that does it. I'll attach that diff to this ticket.

[Thomas Güttler]

I updated the patch to contain your and mine changes. I think it is ready for commit. Karen, can you please test it. If it works for you,
please set the Triage Stage to 'ready for commit'.

I modified your solution only a little bit. The parameters get converted to unicode in unicode, not in init.

[Karen Tracey <kmtracey@…>]

OK, I re-tested with the updated patch (django_template_debug_handle_unicode_error.diff) and it works to fix the problem. The force_unicode in the arg to TemplateSyntaxError seems redundant for this case, since VariableDoesNotExist now will return unicode, but I suppose it is to handle cases where 'e' is something other than VariableDoesNotExist?

[Malcolm Tredinnick]

The only addition I made to this patch before applying it was to also add back the __str__ method (it calls __unicode__). This retains backwards compatibility (plus, it's just polite to always define __str__).

[Malcolm Tredinnick]

(In [7260]) Fixed #5884 -- Added better display error handling for template variables.
Combined work from Karen Tracey and Thomas Güttler.