Memcached backend's get_many() fail with single-use iterators

[guyon.moree@…]

When the keys parameter to get_many() is an iterator, its values will be consumed in a list comprehension. However, later the already-consumed iterator is then passed to zip.

​https://github.com/django/django/blob/master/django/core/cache/backends/memcached.py#L90

This causes a very confusing KeyError which is raised when the cache backend attempts to map lower-level Memcache-backend cache keys back to higher-level Djanco cache keys.

Traceback (most recent call last):
  File "[snip]/django/tests/cache/tests.py", line 1345, in test_get_many_accepts_iterator
    values = cache.get_many(iter(['fizz']))
  File "[snip]/django/lib/python3.6/site-packages/Django-2.2.dev20180706090656-py3.6.egg/django/core/cache/backends/memcached.py", line 91, in get_many
    return {m[k]: v for k, v in ret.items()}
  File "[snip]/django/lib/python3.6/site-packages/Django-2.2.dev20180706090656-py3.6.egg/django/core/cache/backends/memcached.py", line 91, in <dictcomp>
    return {m[k]: v for k, v in ret.items()}
KeyError: ':1:fizz'

[Baptiste Mispelon]

Hi,

The code example shows the expected behavior of a generator: once you iterate over it, it's empty.

Can you describe the actual bug you're encountering in your django application?

The commit that added the line you linked to [1] was already included in 1.3 so your problem must be elsewhere.

I'm going to mark this as needsinfo. Please re-open the ticket with an example on how you trigger the issue from django.

Thanks.

[1] ​https://github.com/django/django/commit/99d247f4cb0c22d19a4482a72a7a93584a5189da

[anonymous]

Ah, I misunderstood the problem myself, makes sense now. It worked in a previous version because the generator was not exhausted there.

thanks a lot, makes sense now

[Christian Barcenas]

Re-opening this ticket, and updating the description with more details. The Memcache backend's get_many() function definitely fails when the key inputs are iterables which can be consumed only once, such as generators.

[Christian Barcenas]

A patch is ready for review on Github: ​https://github.com/django/django/pull/10136

[Tim Graham <timograham@…>]

In c9c6c16:

Fixed #20584 -- Fixed memcached's get_many() with single-use iterators.

Thanks Guyon Morée for the report.