#26600 closed Bug (wontfix)
map says a queryset is not iterable
Reported by: | ihucos | Owned by: | nobody |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | 1.11 |
Severity: | Normal | Keywords: | queryset iterator map |
Cc: | Sergey Fedoseev | Triage Stage: | Unreviewed |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
So I got a funny bug with iterators map and Django that I can not localize or reproduce.
In our production server we get this sometimes, mainly for only one customer and it only fails on a very small percentage of all requests.
Sorry for only chunks of information, not sure if it is enough to process this bug request:
Th error message:
TypeError: argument 2 to map() must support iteration
Chunk of the code where this happens:
if filter( lambda obj: self.pk in verbose_map(attrgetter('pk'), obj.resources.all()) and obj.begins < end and obj.ends > dt, resource_override_qs): return 0
Trying to reconstruct resource_override_qs I come up with:
facility = Facility.objects.\ prefetch_related('resources', 'opening_hours', 'openinghour_overrides', 'slow_hours').\ get(pk=facility.pk) resource_override_qs = facility.facilityresourceoverride_set.prefetch_related('resources')
What I did was to replace map with verbose_map in our production environment.
def verbose_map(function, iterable): try: return map(function, iterable) except TypeError, exc: raise TypeError('map failed: "{}". iterable: {}, type: {}, attrs: {}'.format( # NOQA exc, iterable, type(iterable), dir(iterable)))
After two weeks or so the error occurred again:
map failed: "argument 2 to map() must support iteration". iterable: [<FacilityResource: 4>, <FacilityResource: 5>, <FacilityResource: 6>, <FacilityResource: 7>], type: <class 'django.db.models.query.QuerySet'>, attrs: ['__and__', '__bool__', '__class__', '__deepcopy__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__getstate__', '__hash__', '__init__', '__iter__', '__len__', '__module__', '__new__', '__nonzero__', '__or__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_add_hints', '_as_sql', '_batched_insert', '_clone', '_create_object_from_params', '_db', '_earliest_or_latest', '_extract_model_params', '_fetch_all', '_filter_or_exclude', '_for_write', '_has_filters', '_hints', '_insert', '_known_related_objects', '_merge_known_related_objects', '_merge_sanity_check', '_next_is_sticky', '_populate_pk_values', '_prefetch_done', '_prefetch_related_lookups', '_prefetch_related_objects', '_prepare', '_raw_delete', '_result_cache', '_setup_aggregate_query', '_sticky_filter', '_update', 'aggregate', 'all', 'annotate', 'as_manager', 'bulk_create', 'complex_filter', 'count', 'create', 'dates', 'datetimes', 'db', 'defer', 'delete', 'distinct', 'earliest', 'exclude', 'exists', 'extra', 'filter', 'first', 'get', 'get_or_create', 'in_bulk', 'is_compatible_query_object_type', 'iterator', 'last', 'latest', 'model', 'none', 'only', 'order_by', 'ordered', 'prefetch_related', 'query', 'raw', 'reverse', 'select_for_update', 'select_related', 'update', 'update_or_create', 'using', 'value_annotation', 'values', 'values_list']
As we can see map really gets an QuerySet object, that queryset object does reveal its content in str but python's built in map says its not iterable nevertheless
Alone for the customer where this happens the most that part of the code should be called at least a couple of times per day.
I am going with a workaround like calling list() on the queryset before mapping it or so.
But my company should be ok, to try this again with another verbose_map that may provide more debugging information. My efforts to reproduce this in our code base failed.
Change History (13)
comment:1 by , 8 years ago
Resolution: | → needsinfo |
---|---|
Status: | new → closed |
comment:2 by , 8 years ago
FWIW, the original traceback (which your verbose_map()
hides) may be much more informative. Guessing that you're on Python 2, you may want to rewrite it as something like:
def verbose_map(function, iterable): try: return map(function, iterable) except TypeError: e_cls, e_inst, traceback = sys.exc_info() raise TypeError, ( 'map failed: "{}". iterable: {}, type: {}, attrs: {}'. format(e_inst, iterable, type(iterable), dir(iterable)) ), traceback
comment:3 by , 8 years ago
We got where it comes from.
This exception occurs after Heroku times out the request, but apparently it does not kill it: https://devcenter.heroku.com/articles/request-timeout
So this happens in some funny Heroku environment that I really can't reproduce.
comment:4 by , 7 years ago
I am also seeing this issue, and have to force the queryset into a list to workaround this error. Any plausible idea why this might be happening?
comment:5 by , 7 years ago
Cc: | added |
---|
comment:6 by , 6 years ago
Version: | 1.8 → 1.11 |
---|
I found a way to reproduce the error.
Class A is a simplified version of QuerySet
class A(object): def __init__(self): self._result_cache = None def _fetch_all(self): if self._result_cache is None: self._result_cache = list(self.iterator()) def iterator(self): for x in range(10): if x == 5: raise MemoryError # the type of exception doesn't matter yield x def __iter__(self): self._fetch_all() return iter(self._result_cache)
In [2]: map(str, A()) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/user/workspace/project/service/models.pyc in <module>() ----> 1 map(str, A()) TypeError: argument 2 to map() must support iteration
When using python 3:
In [6]: map(str, A()) --------------------------------------------------------------------------- MemoryError Traceback (most recent call last) <ipython-input-6-aad9944017be> in <module>() ----> 1 map(str, A()) <ipython-input-5-4a8af6429c4d> in __iter__(self) 15 16 def __iter__(self): ---> 17 self._fetch_all() 18 return iter(self._result_cache) 19 <ipython-input-5-4a8af6429c4d> in _fetch_all(self) 6 def _fetch_all(self): 7 if self._result_cache is None: ----> 8 self._result_cache = list(self.iterator()) 9 10 def iterator(self): <ipython-input-5-4a8af6429c4d> in iterator(self) 11 for x in range(10): 12 if x == 5: ---> 13 raise MemoryError 14 yield x 15 MemoryError:
comment:7 by , 6 years ago
Resolution: | needsinfo |
---|---|
Status: | closed → new |
I'm going to reopen so we review the reproduce.
follow-ups: 10 13 comment:8 by , 6 years ago
Resolution: | → wontfix |
---|---|
Status: | new → closed |
I think we can close as wontfix given map
doesn't turn __iter__()
exceptions into TypeError
on Py3k and we don't support Py2k anymore.
Thanks for the investigation Vitaliy.
comment:9 by , 6 years ago
There are at least two ways to workaround the error on Py2k:
- force the queryset into a list when using
map
- use
imap
(fromitertools
) instead ofmap
comment:10 by , 6 years ago
The problem is not only with the map. We faced the same issue when using code like this:
if model_instance in model.objects.all(): ...
Replying to Simon Charette:
I think we can close as wontfix given
map
doesn't turn__iter__()
exceptions intoTypeError
on Py3k and we don't support Py2k anymore.
Thanks for the investigation Vitaliy.
follow-up: 12 comment:11 by , 6 years ago
Philipp, if you hit a similar exception that means map
must be involved somehow.
If you didn't call it yourself then it's likely caused by a usage of map
internally in the queryset iteration code.
In this case the ticket resolution still stands, Python 2 map
's shadowing of the underlying exception makes it impossible to determine its true nature and thus we cannot conclude Django is at fault. As mentioned previously this should all be solved on Python 3 where map
doesn't exhibit this behaviour.
By the way an in
operation on a queryset you are disposing off is likely to trigger a MemoryError
just like in the original report. I'd suggest you opt for something along queryset.filter(pk=instance.pk).exists()
instead.
comment:12 by , 6 years ago
I'm really sorry, of course in case of in
exception message is argument of type 'QuerySet' is not iterable
. This has nothing to do with map
and ofc there's an obvious workaround with exists()
. What I am afraid of is if there are more of this hidden cases where queryset's __iter__
raises an exception and it's shadowed by the outer code exception.
Replying to Simon Charette:
Philipp, if you hit a similar exception that means
map
must be involved somehow.
If you didn't call it yourself then it's likely caused by a usage of
map
internally in the queryset iteration code.
In this case the ticket resolution still stands, Python 2
map
's shadowing of the underlying exception makes it impossible to determine its true nature and thus we cannot conclude Django is at fault. As mentioned previously this should all be solved on Python 3 wheremap
doesn't exhibit this behaviour.
By the way an
in
operation on a queryset you are disposing off is likely to trigger aMemoryError
just like in the original report. I'd suggest you opt for something alongqueryset.filter(pk=instance.pk).exists()
instead.
comment:13 by , 5 years ago
I'm unsure if bumping this up is against forum rules. However, I get this Error in Py3k in production. I'm not using map
in my code, though I'm unsure if Django is internally using it.
This is the code block that produces the error. The save method is overriding the save method of a Mode called Transaction
.
def save(self, *args, **kwargs): obj = self if not obj.ref_number: # Set the reference number before saving transactions = Transaction.objects.values_list("ref_number", flat=True) # cache the list while True: ref_number = random.randint(1000000001, 9999999999) if ref_number not in transactions: obj.ref_number = ref_number super(Transaction, obj).save(*args, **kwargs) return else: super(Transaction, obj).save(*args, **kwargs)
The erroring line of code is: if ref_number not in transactions:
This piece of code was working fine until we had modified another unrelated part of the source code.
items = OrderItem.objects.prefetch_related('toppings').select_related('item').select_related('item__category') order = Order.objects.filter(uuid=order_uuid).prefetch_related( Prefetch( lookup='items', queryset=items ) ).select_related('outlet').select_related('user').select_related('outlet__group') \ .select_for_update(of=('self', 'outlet'))
What we did was add the call to select_for_update
, followed by a transaction which modifies the Order
and Outlet
model. Here, the Order
model is a child model of the Transaction
model.
My guess as to what the problem is is that select_for_update
locks the relevant rows/table which results in the line of code in the save
method to fail internally, ultimately being caught as a cryptic type error. I'll be modifying
transactions = Transaction.objects.values_list("ref_number", flat=True)
to
transactions = Transaction.objects.values_list("ref_number", flat=True).list()
to see if that'll fix the problem.
PS: I'm sorry if my formatting/post/bump is against norms. I'm new here, with this being my first comment.
Replying to Simon Charette:
I think we can close as wontfix given
map
doesn't turn__iter__()
exceptions intoTypeError
on Py3k and we don't support Py2k anymore.
Thanks for the investigation Vitaliy.
No, I can't figure out if or where a bug might in Django based on the report.