Hashing list in Q objects when using __in lookup

[rhyre]

When using djangorestframework with django 2.0.X with lookup _in like id__in=[1,2,3] will raise: unhashable type: 'list'

traceback ends to this file:

/django/utils/tree.py" in __hash__
  74.         return hash((self.__class__, self.connector, self.negated) + tuple(self.children))

When using tuples with __in everything works fine. But when using .values_list() on query I need to use tuple(query) to use it in Q object with __in lookup.
Code worked before with 1.11 version just fine. Also when using _in without Q() everything seems to work just fine.

[Tim Graham]

Please include a minimal example to reproduce the crash. My guess of Model.objects.filter(Q(id__in=[1,2,3])) didn't crash. Thanks.

[rhyre]

Model.objects.prefetch_related('types').exclude(status=1).annotate(
            category=Case(
                When(Q(status__in=[1,2,3]) & Q(id__in=[1,2,3]), then=Value(2)),
                default=Value(1),
                output_field=IntegerField()
            )
        ).only('category')

then it is used in viewsets.ReadOnlyModelViewSet of rest framework in get_queryset.
When using

When(status__in=[1,2,3], id__in=[1,2,3], then=Value(2)),

everything works fine. Also when using Q with tuple as:

When(Q(status__in=(1,2,3)) & Q(id__in=(1,2,3)), then=Value(2)),

its ok.
Only with list it will crash. I've workarounded this to not use Q but it wasn't a problem before.

Without DRF it seems to work just fine. There is a problem with serialization. As it was working before I think its an issue on Django side.

[Carlton Gibson]

Hi. Thanks for the follow-up.

This still isn’t really a minimal reproduce.

Can you provide a test-case that demonstrates the issue? Or a small project?

Any chance you could use git bisect to identify the commit which broke your usage?

Without DRF it seems to work just fine. There is a problem with serialization

What exactly do you mean here? It’s not quite clear. (Do you mean that the ORM call works fine until embedded with the DRF request-response handling? If so you’ll need to provide a sample project I think…(?))

Could you paste the entire traceback for the error?

[Mariusz Felisiak]

It is probably a regression introduced by regression fix (see ​https://github.com/django/django/commit/fc6528b25ab1834be1a478b405bf8f7ec5cf860c).

[rhyre]

Replying to felixxm:

It is probably a regression introduced by regression fix (see ​https://github.com/django/django/commit/fc6528b25ab1834be1a478b405bf8f7ec5cf860c).

In 2.0.8 still not working. Haven't tried 2.1 yet. Too much to refactor to use 2.1.
I'll try to create a demo when I'll have some time.

[Tim Graham]

Crash can be reproduce at the Node level with hash(Node([['a', 1], ['b', 2]])).

[rhyre]

Full traceback:

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  35.             response = get_response(request)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  128.                 response = self.process_exception_by_middleware(e, request)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/rest_framework/viewsets.py" in view
  95.             return self.dispatch(request, *args, **kwargs)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  494.             response = self.handle_exception(exc)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/rest_framework/views.py" in handle_exception
  454.             self.raise_uncaught_exception(exc)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  491.             response = handler(request, *args, **kwargs)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/rest_framework/mixins.py" in list
  40.         queryset = self.filter_queryset(self.get_queryset())

File "/home/rhyre/-/testline_rest.py" in get_queryset
  109.         return LegacyTestlineService.list_testlines()

File "/home/rhyre/-/testline.py" in list_testlines
  162.         return cache_query(key='testline/list', func=query, args=['id', 'name', 'status', 'add_date'], timeout=600)

File "/home/rhyre/-/cache.py" in cache_query
  40.         cache.set(key, dill.dumps(result), timeout)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/dill/_dill.py" in dumps
  293.     dump(obj, file, protocol, byref, fmode, recurse)#, strictio)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/dill/_dill.py" in dump
  286.         pik.dump(obj)

File "/usr/lib64/python3.6/pickle.py" in dump
  409.         self.save(obj)

File "/usr/lib64/python3.6/pickle.py" in save
  496.                 rv = reduce(self.proto)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/query.py" in __getstate__
  224.         self._fetch_all()

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/query.py" in _fetch_all
  1179.             self._result_cache = list(self._iterable_class(self))

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/query.py" in __iter__
  53.         results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in execute_sql
  1055.             sql, params = self.as_sql()

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in as_sql
  448.             extra_select, order_by, group_by = self.pre_sql_setup()

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in pre_sql_setup
  55.         group_by = self.get_group_by(self.select + extra_select, order_by)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in get_group_by
  129.         expressions = self.collapse_group_by(expressions, having_group_by)

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in collapse_group_by
  180.                 expr for expr in expressions if expr in pks or getattr(expr, 'alias', None) not in aliases

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in <listcomp>
  180.                 expr for expr in expressions if expr in pks or getattr(expr, 'alias', None) not in aliases

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/expressions.py" in __hash__
  390.             for key, value in kwargs.items()

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/db/models/expressions.py" in __hash__
  390.             for key, value in kwargs.items()

File "/home/rhyre/.virtualenvs/py3new/lib/python3.6/site-packages/django/utils/tree.py" in __hash__
  74.         return hash((self.__class__, self.connector, self.negated) + tuple(self.children))

Exception Type: TypeError at /testline/list/ajax
Exception Value: unhashable type: 'list'
Request information:
USER: AnonymousUser

[Mariusz Felisiak]

​PR

[GitHub <noreply@…>]

In 9fee2298:

Fixed #29643 -- Fixed crash when combining Q objects with in lookups and lists.

Regression in fc6528b25ab1834be1a478b405bf8f7ec5cf860c.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 2bf766ce:

[2.1.x] Fixed #29643 -- Fixed crash when combining Q objects with in lookups and lists.

Regression in fc6528b25ab1834be1a478b405bf8f7ec5cf860c.
Backport of 9fee229874367beafd532dad6d0f9ff9676ded0b from master

[Tim Graham]

This didn't completely solve the issue, see #29838 for follow up.