Opened 4 years ago

Closed 14 months ago

Last modified 12 months ago

#16335 closed Bug (fixed)

Document why you cannot iterate defaultdict in templates

Reported by: jacob.ninja.dev@… Owned by: bluejeansummer
Component: Documentation Version: 1.3
Severity: Normal Keywords:
Cc: bluejeansummer, akvadrako Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: yes UI/UX: no

Description

I am unable to iterate a defaultdict using a for loop in a template.

Code to reproduce issue:

dictionary = defaultdict(list)
dictionary['foo'].append('bar')

{% for key, value in dictionary.items %}
  {# Never iterates #}
{% endfor %}

Attachments (2)

defaultdict_conversion.diff (1.6 KB) - added by bluejeansummer 4 years ago.
Convert defaultdicts to dicts before template lookups.
defaultdict_docs.diff (935 bytes) - added by bluejeansummer 4 years ago.
Add example of current template behavior w.r.t. defaultdicts.

Download all attachments as: .zip

Change History (21)

comment:1 Changed 4 years ago by mattmcc

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Right, because dictionary key access takes precedence in template variable lookups, so {% ... dictionary.items %} adds an empty list named 'items' to the dictionary. Not sure there's a good workaround here.

comment:2 Changed 4 years ago by aaugustin

  • Triage Stage changed from Unreviewed to Accepted

Indeed, it boils down to the fact that the template language uses the same syntax for dictionary and attribute lookups. The resolution order is documented here: https://docs.djangoproject.com/en/1.3/topics/templates/#variables. It's probably a bad idea to change it.

We could add a note there to warn about objects that accept a dictionary lookup with any name, suggesting to convert them to dict before passing them to the view, in your example:

context['dictionary'] = dict(dictionary)

comment:3 Changed 4 years ago by bluejeansummer

  • Cc bluejeansummer added
  • Needs documentation set
  • Needs tests set
  • Owner changed from nobody to bluejeansummer
  • Status changed from new to assigned

How much would break if we were to do the defaultdict -> dict conversion automatically?

Changed 4 years ago by bluejeansummer

Convert defaultdicts to dicts before template lookups.

comment:4 Changed 4 years ago by bluejeansummer

  • Has patch set
  • Needs tests unset
  • Patch needs improvement set

Changed 4 years ago by bluejeansummer

Add example of current template behavior w.r.t. defaultdicts.

comment:5 Changed 4 years ago by bluejeansummer

  • Needs documentation unset

After some consideration, perhaps performing the conversion is not such a good idea. I've added an example to the docs that explains why the current behavior happens.

comment:6 Changed 4 years ago by akvadrako

  • Cc akvadrako added

comment:7 Changed 3 years ago by oinopion

  • Component changed from Template system to Documentation
  • Triage Stage changed from Accepted to Ready for checkin

Patch with docs explains issue nicely.

comment:8 Changed 3 years ago by anonymous

Patch looks great, however just to be super clear, I think it should include a bit like, "So while this looks like it should do defaultdict.iteritems() in fact it's doing defaultdictiteritems? which returns the default value".

comment:9 Changed 3 years ago by SmileyChris

Hrm, I wonder if the _resolve_lookup method could be made smarter with something like the following for the dictionary lookup section:

try:
    if bit not in current:
        raise KeyError
except TypeError:
    pass
current = current[bit]

comment:10 Changed 3 years ago by Aymeric Augustin

  • Resolution set to fixed
  • Status changed from assigned to closed

Fixed #16335 -- Clarified an unintuitive behavior.

The DTL will perform dict lookup before method lookup, which yields
an unexpected result for defaultdicts.

Changeset: d171b3cc0b32374fd5891254b04693e9ec8ed946

comment:11 Changed 3 years ago by Aymeric Augustin

Fixed #16335 -- Clarified an unintuitive behavior.

The DTL will perform dict lookup before method lookup, which yields
an unexpected result for defaultdicts.

Changeset: d171b3cc0b32374fd5891254b04693e9ec8ed946

comment:12 Changed 3 years ago by Aymeric Augustin

Fixed #16335 -- Clarified an unintuitive behavior.

The DTL will perform dict lookup before method lookup, which yields
an unexpected result for defaultdicts.

Changeset: d171b3cc0b32374fd5891254b04693e9ec8ed946

comment:13 Changed 2 years ago by anonymous

Having this case:

my_dict = collections.defaultdict(lambda: collections.defaultdict(list))    
dictionary['foo']['foo1'].append(obj)

and doing

return Response({'my_dict ': dict(my_dict)}, template_name='_internal_template.html')

I'm able to loop the first for only

{% for groups in my_dict.itervalues  %}
    groups = {{ groups }} <br><br> //It prints: groups = defaultdict(<type 'list'>, {1: [<Obj: Obj 1 by me>]})  
    {% for cols in groups.itervalues  %}
      cols = {{ cols }} <br><br>
      {% for maps in cols.itervalues  %}

comment:14 Changed 17 months ago by krommail@…

  • Resolution fixed deleted

Django 1.6, Pyhton 3.3.3. Even in the simpliest case:
{% for key, values in dict.items %}

{{ key }}

{% endfor %}
there is nothing in "key".

While
{% for key in dict %}

{{ key }}

{% endfor %}

works well.
Also preliminary conversion of a defaultdict to dict helps.

comment:15 Changed 14 months ago by ericpulvino@…

I am also seeing this for Django 1.6.1 python 2.7.6. Conversion to standard dict does not help in my case. Otherwise same symptoms as the post from krommail.

comment:16 Changed 14 months ago by ericpulvino@…

After upgrading to 1.6.5 still with python 2.7.6 the workaround (converting to standard dictionary prior to handing dict to the template) functions.

As a Django newbie I have sunk 5 hours on this silly issue. I need a break today.

comment:17 Changed 14 months ago by anonymous

  • Status changed from closed to new

comment:18 Changed 14 months ago by timo

  • Resolution set to fixed
  • Status changed from new to closed
  • Summary changed from Cannot iterate defaultdict in template to Document why you cannot iterate defaultdict in templates

The documentation that was added as part of the ticket [d171b3cc0b32374fd5891254b04693e9ec8ed946] explains why it doesn't work.

comment:19 Changed 12 months ago by diego.gaustein@…

I am actually getting a MemoryError (which sometimes end in segfault) when trying to iterate the values in a defaultdict

In views.py:

totals = defaultdict(Decimal)
add_stuff_to_it(totals)

In the template:

    {% for t in totals.values %}
    {{ t }}
    {% endfor %}

Results in

$ python manage.py runserver 0.0.0.0:8000 --settings=xxx.settings_debug
Validating models...

0 errors found
July 30, 2014 - 09:57:05
Django version 1.6.5, using settings 'xxx.settings_debug'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 67, in __call__
    return self.application(environ, start_response)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 206, in __call__
    response = self.get_response(request)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 194, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 229, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/views/debug.py", line 69, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/views/debug.py", line 324, in get_traceback_html
    return t.render(c)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 305, in render
    return nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 36, in render
    output = self.nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 212, in render
    return nodelist.render(context)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    return mark_safe(''.join(bits))
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/django/utils/safestring.py", line 116, in mark_safe
    return SafeText(s)
MemoryError
[30/Jul/2014 09:57:16] "GET /statistics/order_book_and_wip/ HTTP/1.1" 500 59

Strangely, iterating over the keys seems to work fine.
Setting the default_factory to None as detailed here makes it work as expected. It would seem like a cheap and sensible workaround to apply that to defaultdicts. Or otherwise at least raise an exception so users can know what's going on.

Note: See TracTickets for help on using tickets.
Back to Top