Opened 5 years ago

Closed 2 years ago

#24525 closed Bug (fixed)

AssertionError at `Query.change_aliases`

Reported by: coolRR Owned by: nobody
Component: Database layer (models, ORM) Version: 1.11
Severity: Release blocker Keywords:
Cc: kevmitch Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description (last modified by coolRR)

(Django version 1.7.7)

I'm getting an assertion error on the first line of Query.change_aliases:

        assert set(change_map.keys()).intersection(set(change_map.values())) == set()

Problem is, I can't post more data, because the queryset has confidential client information :(

I can post the value of change_map, with identifying details renamed:

    {'T5': 'T10',
     'T6': 'T11',
     'T8': 'T13',
     'my_app_follow': 'my_app_follow',
     'my_app_stack__readers': 'T7',
     'my_app_stack__writers': 'T9',
     'my_app_user': 'T8'}

This happened when doing & on two querysets, one of which having .distinct() applied on it, among other things.

Attachments (1)

and_assertion_error.zip (11.2 KB) - added by kevmitch 4 years ago.
minimal example project exhibiting the error

Download all attachments as: .zip

Change History (22)

comment:1 Changed 5 years ago by coolRR

Description: modified (diff)

comment:2 Changed 5 years ago by Tim Graham

Couldn't you change the names of the models and values of the data and offer a test case to reproduce the error?

comment:3 Changed 5 years ago by coolRR

These querysets are passed through several functions in different files so it'll be hard to construct them in straight code. It'll be about an hour of work for me which is too much.

comment:4 Changed 5 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

Well, trying to reproduce this from the provided details will likely take much longer than an hour.

comment:5 Changed 5 years ago by Tim Graham

Resolution: needsinfo
Status: closednew

Reporter provided the SQL from the queries that are being ANDed. Maybe this is enough to be helpful to an ORM expert; reopening on that basis.

SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created", "soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name", "soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading", "soufle_app_stack"."is_discussion_stack", "soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN "soufle_app_stack__writers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__writers"."stack_id" ) WHERE ("soufle_app_stack"."datetime_canceled" IS NULL AND ("soufle_app_stack"."is_public_reading" = True OR "soufle_app_stack__readers"."user_id" = 147 OR "soufle_app_stack__writers"."user_id" = 147 OR "soufle_app_stack"."owner_id" = 147)) ORDER BY "soufle_app_stack"."datetime_created" DESC

SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created", "soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name", "soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading", "soufle_app_stack"."is_discussion_stack", "soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN "soufle_app_stack__writers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__writers"."stack_id" ) INNER JOIN "soufle_app_follow" ON ( "soufle_app_stack"."id" = "soufle_app_follow"."stack_id" ) WHERE ("soufle_app_stack"."datetime_canceled" IS NULL AND ("soufle_app_stack"."is_public_reading" = True OR "soufle_app_stack__readers"."user_id" = 147 OR "soufle_app_stack__writers"."user_id" = 147 OR "soufle_app_stack"."owner_id" = 147) AND NOT ("soufle_app_stack"."id" IN (SELECT U1."private_stack_id" AS "private_stack_id" FROM "soufle_app_user" U1 WHERE (U1."id" = 147 AND U1."private_stack_id" IS NOT NULL))) AND NOT ("soufle_app_stack"."id" IN (SELECT U1."public_stack_id" AS "public_stack_id" FROM "soufle_app_user" U1 WHERE (U1."id" = 147 AND U1."public_stack_id" IS NOT NULL))) AND "soufle_app_follow"."user_id" = 147) ORDER BY "soufle_app_stack"."datetime_created" DESC

comment:6 Changed 5 years ago by coolRR

Can anyone possibly help with this? I'm stuck on this problem and I don't know how to solve it.

comment:7 Changed 5 years ago by Josh Smeaton

Can you try your queryset out with django 1.8 please? I've tried to emulate the idea by combining two querysets, one that has distinct applied, and that behaviour is not allowed (in 1.8):

In [9]: Company.objects.filter(name='HI').distinct() & Company.objects.exclude(name='HI')
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-9-b0c9de44f4df> in <module>()
----> 1 Company.objects.filter(name='HI').distinct() & Company.objects.exclude(name='HI')

/Users/smeatonj/Development/django/django/db/models/query.py in __and__(self, other)
    305         combined = self._clone()
    306         combined._merge_known_related_objects(other)
--> 307         combined.query.combine(other.query, sql.AND)
    308         return combined
    309

/Users/smeatonj/Development/django/django/db/models/sql/query.py in combine(self, rhs, connector)
    494             "Cannot combine queries once a slice has been taken."
    495         assert self.distinct == rhs.distinct, \
--> 496             "Cannot combine a unique query with a non-unique query."
    497         assert self.distinct_fields == rhs.distinct_fields, \
    498             "Cannot combine queries with different distinct fields."

AssertionError: Cannot combine a unique query with a non-unique query.

So if you hit the same problem as I did in 1.8, then there's little motivation to try to investigate the cause of your original problem in 1.7 since it's explicitly disallowed in 1.8. I would venture that the reason it's disallowed in 1.8 (assuming it was "valid" in 1.7) is for similar reasons that you're seeing now.

The SQL dump provided above doesn't appear to show any distinct clauses, but your original problem description did. We don't have enough information here to figure out the problem.

If you could at least provide the stacktrace when the assert is hit, that might help narrow it down. But, honestly, if this is important enough to your application then you should spend the time needed to create a reproducible series of steps. We help where we can, but I think you'd be hard pressed to find someone willing to try combination after combination trying to arrive at a similar situation that you're in because it'd take you "too much time".

comment:8 in reply to:  7 Changed 5 years ago by coolRR

Hi Josh,

Thanks for your help.

  • Turns out I was wrong about the .distinct thing, it's to be applied after the failing code and not before it, so it's not relevant. That was my mistake. In other words .distinct is definitely not used in the failing query, so we can't expect to get the "Cannot combine a unique query with a non-unique query" error.
  • I tried to run my code on Django 1.8, and got the exact same error.
  • Here is a traceback of the error: http://i.imgur.com/nM748ne.jpg The frame just above the one visible is the frame in which I do & between two querysets. (Didn't include these frames because they have client code.) (This traceback was done on Django 1.8, if you want the same on Django 1.7.7 I can do that too.) If you'll want me to expand any of the "Local vars" sections, let me know.

Also, if you're interested enough and have the time to do a VNC or TeamViewer session to my dev machine, I can show you the whole thing. Of course, I'll totally understand if you're too busy. If so I'll try to do a sample app. (Trying to avoid billing my client for the hours that this will take.)

Thanks,
Ram.

comment:9 Changed 5 years ago by Josh Smeaton

It's not likely that just seeing the problem is going to be enough to reproduce on my own. I sympathise with your situation, but you're trying to avoid billing your client by palming the work off for free to open source contributors. We're not tier 3 support. You should (IMO) be billing for bug fixes in situations like this.

I'm happy to meet you half way. If you can put a test case together then we can work on fixing the issue.

Cheers

comment:10 Changed 5 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

Changed 4 years ago by kevmitch

Attachment: and_assertion_error.zip added

minimal example project exhibiting the error

comment:11 Changed 4 years ago by kevmitch

In Django 1.8, with the models

from django.db import models

class Year(models.Model):
    date = models.DateField(auto_now_add=True)

class Student(models.Model):
    year    = models.ForeignKey('Year')
    number  = models.PositiveIntegerField()
    courses = models.ManyToManyField('Course')

class Course(models.Model):
    year       = models.ForeignKey('Year')
    department = models.CharField(max_length=4)
    number     = models.PositiveIntegerField()

the following

course_instance.student_set.all() & year_instance.student_set.exclude(courses__in=[other_course_instance])

will raise

Traceback (most recent call last):
  File "/home/kevmitch/projects/apprentice/and_assertion_error/enrollment/tests.py", line 24, in test_and
    result = math101.student_set.all() & year.student_set.exclude(courses__in=[anth100])
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 211, in __and__
    combined.query.combine(other.query, sql.AND)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 556, in combine
    w.relabel_aliases(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 285, in relabel_aliases
    child.relabel_aliases(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 287, in relabel_aliases
    self.children[pos] = child.relabeled_clone(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/lookups.py", line 185, in relabeled_clone
    new.rhs = new.rhs.relabeled_clone(relabels)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 329, in relabeled_clone
    clone.change_aliases(change_map)
  File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 787, in change_aliases
    assert set(change_map.keys()).intersection(set(change_map.values())) == set()
AssertionError

The tests of attached project demonstrates it in action.

Essential elements for reproduction:

  • One of the querysets generated from related manager of the ManyToMany target
  • The other queryset is generated by exclude()ing on the ManyToManyField from the related manager of a "versioning" ForeignKey model

comment:12 Changed 4 years ago by kevmitch

Resolution: needsinfo
Status: closednew

comment:13 Changed 4 years ago by kevmitch

Cc: kevmitch added

comment:14 Changed 4 years ago by Tim Graham

Severity: NormalRelease blocker
Triage Stage: UnreviewedAccepted

comment:15 Changed 4 years ago by Tim Graham

Has patch: set

Removing the assertion seems to work. Is there a better solution?

comment:16 Changed 4 years ago by Tim Graham

Patch needs improvement: set

comment:17 Changed 4 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In 2dc9ec56:

Fixed #24525 -- Fixed AssertionError in some complex queries.

Thanks Anssi Kääriäinen for providing the solution.

comment:18 Changed 4 years ago by Tim Graham <timograham@…>

In 62347208:

[1.8.x] Fixed #24525 -- Fixed AssertionError in some complex queries.

Thanks Anssi Kääriäinen for providing the solution.

Backport of 2dc9ec5616a942de3a0886a707f93988f56dd594 from master

comment:19 Changed 2 years ago by Christian Karrié

Resolution: fixed
Status: closednew

Hi all, unfortunately I have to reopen this issue.

After migrating from Django 1.10.6 to 1.11.9 the error reoccurs:

Code:

qs_out = qs.filter(*out_filters)
qs_in = qs.filter(*in_filters)
qs_merged = qs_in | qs_out

Vars:

in_filters = [<Q: (NOT (AND: ('contract__customer_id', 1)))>, <Q: (NOT (AND: ('contract__category__deletion', True)))>, <Q: (NOT (AND: ('contract__date_from__lte', datetime.date(2018, 1, 10)), ('contract__category__blocked', True), ('contract__date_until__gte', datetime.date(2018, 1, 10))))>, <Q: (AND: ('id', 0))>]
out_filters = [<Q: (OR: (AND: (NOT (AND: ('contract__source_customer_id', 1))), ('contract__customer_id', 1)), (AND: (NOT (AND: ('contract__source_customer_id', 1))), (NOT (AND: ('contract__customer_id', 1)))))>, <Q: (NOT (AND: ('contract__category__incoming', True)))>]

It seems to fail, if qs_in or qs_out is empty. But I'm not sure.

Printing change_map gives me:

{'T7': u'invoice_customer', u'invoice_customer': 'T7'}

Removing the assert works.

Last edited 2 years ago by Christian Karrié (previous) (diff)

comment:20 Changed 2 years ago by Christian Karrié

Version: 1.71.11

comment:21 Changed 2 years ago by Tim Graham

Resolution: fixed
Status: newclosed

As the fix for this ticket is released, please open a new ticket with complete details to reproduce. Ideally, you could also bisect to find the commit where the behavior changed.

Note: See TracTickets for help on using tickets.
Back to Top