Opened 5 years ago

Closed 2 years ago

Last modified 2 years ago

#16335 closed Bug (fixed)

Document why you cannot iterate defaultdict in templates

Reported by: jacob.ninja.dev@… Owned by: Nick Meharry
Component: Documentation Version: 1.3
Severity: Normal Keywords:
Cc: Nick Meharry, akvadrako Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: yes UI/UX: no

Description

I am unable to iterate a defaultdict using a for loop in a template.

Code to reproduce issue:

dictionary = defaultdict(list)
dictionary['foo'].append('bar')

{% for key, value in dictionary.items %}
  {# Never iterates #}
{% endfor %}

Attachments (2)

defaultdict_conversion.diff (1.6 KB) - added by Nick Meharry 5 years ago.
Convert defaultdicts to dicts before template lookups.
defaultdict_docs.diff (935 bytes) - added by Nick Meharry 5 years ago.
Add example of current template behavior w.r.t. defaultdicts.

Download all attachments as: .zip

Change History (21)

comment:1 Changed 5 years ago by Matt McClanahan

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset

Right, because dictionary key access takes precedence in template variable lookups, so {% ... dictionary.items %} adds an empty list named 'items' to the dictionary. Not sure there's a good workaround here.

comment:2 Changed 5 years ago by Aymeric Augustin

Triage Stage: UnreviewedAccepted

Indeed, it boils down to the fact that the template language uses the same syntax for dictionary and attribute lookups. The resolution order is documented here: https://docs.djangoproject.com/en/1.3/topics/templates/#variables. It's probably a bad idea to change it.

We could add a note there to warn about objects that accept a dictionary lookup with any name, suggesting to convert them to dict before passing them to the view, in your example:

context['dictionary'] = dict(dictionary)

comment:3 Changed 5 years ago by Nick Meharry

Cc: Nick Meharry added
Needs documentation: set
Needs tests: set
Owner: changed from nobody to Nick Meharry
Status: newassigned

How much would break if we were to do the defaultdict -> dict conversion automatically?

Changed 5 years ago by Nick Meharry

Attachment: defaultdict_conversion.diff added

Convert defaultdicts to dicts before template lookups.

comment:4 Changed 5 years ago by Nick Meharry

Has patch: set
Needs tests: unset
Patch needs improvement: set

Changed 5 years ago by Nick Meharry

Attachment: defaultdict_docs.diff added

Add example of current template behavior w.r.t. defaultdicts.

comment:5 Changed 5 years ago by Nick Meharry

Needs documentation: unset

After some consideration, perhaps performing the conversion is not such a good idea. I've added an example to the docs that explains why the current behavior happens.

comment:6 Changed 5 years ago by akvadrako

Cc: akvadrako added

comment:7 Changed 5 years ago by Tomek Paczkowski

Component: Template systemDocumentation
Triage Stage: AcceptedReady for checkin

Patch with docs explains issue nicely.

comment:8 Changed 5 years ago by anonymous

Patch looks great, however just to be super clear, I think it should include a bit like, "So while this looks like it should do defaultdict.iteritems() in fact it's doing defaultdictiteritems? which returns the default value".

comment:9 Changed 5 years ago by Chris Beaven

Hrm, I wonder if the _resolve_lookup method could be made smarter with something like the following for the dictionary lookup section:

try:
    if bit not in current:
        raise KeyError
except TypeError:
    pass
current = current[bit]

comment:10 Changed 4 years ago by Aymeric Augustin

Resolution: fixed
Status: assignedclosed

Fixed #16335 -- Clarified an unintuitive behavior.

The DTL will perform dict lookup before method lookup, which yields
an unexpected result for defaultdicts.

Changeset: d171b3cc0b32374fd5891254b04693e9ec8ed946

comment:11 Changed 4 years ago by Aymeric Augustin

Fixed #16335 -- Clarified an unintuitive behavior.

The DTL will perform dict lookup before method lookup, which yields
an unexpected result for defaultdicts.

Changeset: d171b3cc0b32374fd5891254b04693e9ec8ed946

comment:12 Changed 4 years ago by Aymeric Augustin

Fixed #16335 -- Clarified an unintuitive behavior.

The DTL will perform dict lookup before method lookup, which yields
an unexpected result for defaultdicts.

Changeset: d171b3cc0b32374fd5891254b04693e9ec8ed946

comment:13 Changed 3 years ago by anonymous

Having this case:

my_dict = collections.defaultdict(lambda: collections.defaultdict(list))    
dictionary['foo']['foo1'].append(obj)

and doing

return Response({'my_dict ': dict(my_dict)}, template_name='_internal_template.html')

I'm able to loop the first for only

{% for groups in my_dict.itervalues  %}
    groups = {{ groups }} <br><br> //It prints: groups = defaultdict(<type 'list'>, {1: [<Obj: Obj 1 by me>]})  
    {% for cols in groups.itervalues  %}
      cols = {{ cols }} <br><br>
      {% for maps in cols.itervalues  %}

comment:14 Changed 3 years ago by krommail@…

Resolution: fixed

Django 1.6, Pyhton 3.3.3. Even in the simpliest case:
{% for key, values in dict.items %}

{{ key }}

{% endfor %}
there is nothing in "key".

While
{% for key in dict %}

{{ key }}

{% endfor %}

works well.
Also preliminary conversion of a defaultdict to dict helps.

comment:15 Changed 2 years ago by ericpulvino@…

I am also seeing this for Django 1.6.1 python 2.7.6. Conversion to standard dict does not help in my case. Otherwise same symptoms as the post from krommail.

comment:16 Changed 2 years ago by ericpulvino@…

After upgrading to 1.6.5 still with python 2.7.6 the workaround (converting to standard dictionary prior to handing dict to the template) functions.

As a Django newbie I have sunk 5 hours on this silly issue. I need a break today.

comment:17 Changed 2 years ago by anonymous

Status: closednew

comment:18 Changed 2 years ago by Tim Graham

Resolution: fixed
Status: newclosed
Summary: Cannot iterate defaultdict in templateDocument why you cannot iterate defaultdict in templates

The documentation that was added as part of the ticket [d171b3cc0b32374fd5891254b04693e9ec8ed946] explains why it doesn't work.

comment:19 Changed 2 years ago by diego.gaustein@…

I am actually getting a MemoryError (which sometimes end in segfault) when trying to iterate the values in a defaultdict

In views.py:

totals = defaultdict(Decimal)
add_stuff_to_it(totals)

In the template:

    {% for t in totals.values %}
    {{ t }}
    {% endfor %}

Results in

$ python manage.py runserver 0.0.0.0:8000 --settings=xxx.settings_debug
Validating models...

0 errors found
July 30, 2014 - 09:57:05
Django version 1.6.5, using settings 'xxx.settings_debug'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 67, in __call__
    return self.application(environ, start_response)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 206, in __call__
    response = self.get_response(request)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 194, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 229, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/views/debug.py", line 69, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/views/debug.py", line 324, in get_traceback_html
    return t.render(c)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 305, in render
    return nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 36, in render
    output = self.nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 212, in render
    return nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    return mark_safe(''.join(bits))
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/utils/safestring.py", line 116, in mark_safe
    return SafeText(s)
MemoryError
[30/Jul/2014 09:57:16] "GET /statistics/order_book_and_wip/ HTTP/1.1" 500 59

Strangely, iterating over the keys seems to work fine.
Setting the default_factory to None as detailed here makes it work as expected. It would seem like a cheap and sensible workaround to apply that to defaultdicts. Or otherwise at least raise an exception so users can know what's going on.

Note: See TracTickets for help on using tickets.
Back to Top