TypeError: unhashable type: 'list' when paginating queryset with KeyTransform annotation

[Jaap Roes]

We have encountered a unexpected error when trying to upgrade an application from Django 2.1 to 2.2.

In our application we have the requirement to order a queryset on a value that's found in a nested datastructure stored in a JSON field. To do this we use KeyTransforms to annotate the queryset with this value and order on this annotation. This queryset is then paginated.

This used to work in Django 2.1, but now raises a TypeError in Django 2.2.

I created a minimal example to reproduce the error:

models.py:

from django.db import models
from django.contrib.postgres.fields import JSONField

class Example(models.Model):
    data = JSONField(default=dict)

tests.py:

from django.test import TestCase
from django.core.paginator import Paginator
from django.contrib.postgres.fields.jsonb import KeyTextTransform, KeyTransform

from .models import Example

class TestKeyTransformPagination(TestCase):
    def setUp(self):
        Example.objects.create(data={
            'translations': [
                {'title': 'Ladies and gentleman'},
                {'title': 'Dames en heren'}
            ]
        })
        Example.objects.create(data={
            'translations': [
                {'title': 'Apples and pears'},
                {'title': 'Appels en peren'}
            ]
        })
        # The next example queryset raises: TypeError: unhashable type: 'list'
        # qs = Example.objects.order_by('data__translations__0__title')
        # qs[0]
        #
        # This is a workaround to achieve the desired ordering
        extract_title = KeyTextTransform('title', KeyTransform('0', KeyTransform('translations', 'data')))
        self.qs = Example.objects.annotate(title=extract_title).order_by('title')

    def test_queryset(self):
        self.assertEqual('Apples and pears', self.qs[0].data['translations'][0]['title'])

    def test_pagination(self):
        # Works on 2.1.x and master, raises TypeError: unhashable type: 'list' on 2.2
        paginator = Paginator(self.qs, per_page=10)
        page = paginator.page(1)
        self.assertEqual('Apples and pears', page.object_list[0].data['translations'][0]['title'])

While investigating this issue I discovered that this error does not happen on the current master.

After bisecting I can tell that commit 3767c7ff391d5f277e25bca38ef3730ddf9cea9c (Fixed #29244) introduces (or exposes) the exception and 3f32154f40a855afa063095e3d091ce6be21f2c5 (Fixed #30188) fixes the error.

Is it possible to backport the fix for #30188 to Django 2.2 so we can upgrade our application?

[Jaap Roes]

By the way, I'm not sure if Example.objects.order_by('data__translations__0__title')[0] is supposed to work? It currently raises a TypeError which is why we are using annotate with KeyTransforms as a work around.

[Carlton Gibson]

Thanks for the report. I can reproduce with the provided example. (Passes on 2.1.x, fails on 2.2.x)

[Can Sarıgöl]

Hi, is this can be a solution? ​PR

[Jaap Roes]

Replying to Can Sarıgöl:

Hi, is this can be a solution? ​PR

That looks good, it seems that it would also remove the need for our workaround, which I'm always in favor of :-)

[Mariusz Felisiak]

Related ticket #29139.

[Mariusz Felisiak <felisiak.mariusz@…>]

In d87bd29:

Fixed #30335, #29139 -- Fixed crash when ordering or aggregating over a nested JSONField key transform.

[Mariusz Felisiak <felisiak.mariusz@…>]

In e85317d7:

[2.2.x] Fixed #30335, #29139 -- Fixed crash when ordering or aggregating over a nested JSONField key transform.

Backport of d87bd29c4f8dfcdf3f4a4eb8340e6770a2416fe3 from master.