Opened 7 weeks ago

Closed 5 weeks ago

Last modified 5 weeks ago

#30335 closed Bug (fixed)

TypeError: unhashable type: 'list' when paginating queryset with KeyTransform annotation

Reported by: Jaap Roes Owned by: Can Sarıgöl
Component: Database layer (models, ORM) Version: 2.2
Severity: Release blocker Keywords: postgresql jsonb keytransform annotation pagination
Cc: Can Sarıgöl Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

We have encountered a unexpected error when trying to upgrade an application from Django 2.1 to 2.2.

In our application we have the requirement to order a queryset on a value that's found in a nested datastructure stored in a JSON field. To do this we use KeyTransforms to annotate the queryset with this value and order on this annotation. This queryset is then paginated.

This used to work in Django 2.1, but now raises a TypeError in Django 2.2.

I created a minimal example to reproduce the error:

models.py:

from django.db import models
from django.contrib.postgres.fields import JSONField

class Example(models.Model):
    data = JSONField(default=dict)

tests.py:

from django.test import TestCase
from django.core.paginator import Paginator
from django.contrib.postgres.fields.jsonb import KeyTextTransform, KeyTransform

from .models import Example

class TestKeyTransformPagination(TestCase):
    def setUp(self):
        Example.objects.create(data={
            'translations': [
                {'title': 'Ladies and gentleman'},
                {'title': 'Dames en heren'}
            ]
        })
        Example.objects.create(data={
            'translations': [
                {'title': 'Apples and pears'},
                {'title': 'Appels en peren'}
            ]
        })
        # The next example queryset raises: TypeError: unhashable type: 'list'
        # qs = Example.objects.order_by('data__translations__0__title')
        # qs[0]
        #
        # This is a workaround to achieve the desired ordering
        extract_title = KeyTextTransform('title', KeyTransform('0', KeyTransform('translations', 'data')))
        self.qs = Example.objects.annotate(title=extract_title).order_by('title')

    def test_queryset(self):
        self.assertEqual('Apples and pears', self.qs[0].data['translations'][0]['title'])

    def test_pagination(self):
        # Works on 2.1.x and master, raises TypeError: unhashable type: 'list' on 2.2
        paginator = Paginator(self.qs, per_page=10)
        page = paginator.page(1)
        self.assertEqual('Apples and pears', page.object_list[0].data['translations'][0]['title'])

While investigating this issue I discovered that this error does not happen on the current master.

After bisecting I can tell that commit 3767c7ff391d5f277e25bca38ef3730ddf9cea9c (Fixed #29244) introduces (or exposes) the exception and 3f32154f40a855afa063095e3d091ce6be21f2c5 (Fixed #30188) fixes the error.

Is it possible to backport the fix for #30188 to Django 2.2 so we can upgrade our application?

Change History (9)

comment:1 Changed 7 weeks ago by Jaap Roes

By the way, I'm not sure if Example.objects.order_by('data__translations__0__title')[0] is supposed to work? It currently raises a TypeError which is why we are using annotate with KeyTransforms as a work around.

comment:2 Changed 7 weeks ago by Carlton Gibson

Severity: NormalRelease blocker
Triage Stage: UnreviewedAccepted

Thanks for the report. I can reproduce with the provided example. (Passes on 2.1.x, fails on 2.2.x)

comment:3 Changed 7 weeks ago by Can Sarıgöl

Cc: Can Sarıgöl added
Has patch: set

Hi, is this can be a solution? PR

comment:4 in reply to:  3 Changed 7 weeks ago by Jaap Roes

Replying to Can Sarıgöl:

Hi, is this can be a solution? PR

That looks good, it seems that it would also remove the need for our workaround, which I'm always in favor of :-)

comment:5 Changed 6 weeks ago by felixxm

Related ticket #29139.

comment:6 Changed 6 weeks ago by Tobias Kunze

Owner: changed from nobody to Can Sarıgöl
Status: newassigned

comment:7 Changed 5 weeks ago by felixxm

Triage Stage: AcceptedReady for checkin

comment:8 Changed 5 weeks ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In d87bd29:

Fixed #30335, #29139 -- Fixed crash when ordering or aggregating over a nested JSONField key transform.

comment:9 Changed 5 weeks ago by Mariusz Felisiak <felisiak.mariusz@…>

In e85317d7:

[2.2.x] Fixed #30335, #29139 -- Fixed crash when ordering or aggregating over a nested JSONField key transform.

Backport of d87bd29c4f8dfcdf3f4a4eb8340e6770a2416fe3 from master.

Note: See TracTickets for help on using tickets.
Back to Top