Opened 10 years ago
Closed 7 years ago
#24525 closed Bug (fixed)
AssertionError at `Query.change_aliases`
Reported by: | Ram Rachum | Owned by: | nobody |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | 1.11 |
Severity: | Release blocker | Keywords: | |
Cc: | kevmitch | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | yes |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
(Django version 1.7.7)
I'm getting an assertion error on the first line of Query.change_aliases
:
assert set(change_map.keys()).intersection(set(change_map.values())) == set()
Problem is, I can't post more data, because the queryset has confidential client information :(
I can post the value of change_map
, with identifying details renamed:
{'T5': 'T10', 'T6': 'T11', 'T8': 'T13', 'my_app_follow': 'my_app_follow', 'my_app_stack__readers': 'T7', 'my_app_stack__writers': 'T9', 'my_app_user': 'T8'}
This happened when doing &
on two querysets, one of which having .distinct()
applied on it, among other things.
Attachments (1)
Change History (22)
comment:1 by , 10 years ago
Description: | modified (diff) |
---|
comment:2 by , 10 years ago
comment:3 by , 10 years ago
These querysets are passed through several functions in different files so it'll be hard to construct them in straight code. It'll be about an hour of work for me which is too much.
comment:4 by , 10 years ago
Resolution: | → needsinfo |
---|---|
Status: | new → closed |
Well, trying to reproduce this from the provided details will likely take much longer than an hour.
comment:5 by , 10 years ago
Resolution: | needsinfo |
---|---|
Status: | closed → new |
Reporter provided the SQL from the queries that are being ANDed. Maybe this is enough to be helpful to an ORM expert; reopening on that basis.
SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created", "soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name", "soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading", "soufle_app_stack"."is_discussion_stack", "soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN "soufle_app_stack__writers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__writers"."stack_id" ) WHERE ("soufle_app_stack"."datetime_canceled" IS NULL AND ("soufle_app_stack"."is_public_reading" = True OR "soufle_app_stack__readers"."user_id" = 147 OR "soufle_app_stack__writers"."user_id" = 147 OR "soufle_app_stack"."owner_id" = 147)) ORDER BY "soufle_app_stack"."datetime_created" DESC SELECT "soufle_app_stack"."id", "soufle_app_stack"."datetime_created", "soufle_app_stack"."datetime_updated", "soufle_app_stack"."_name", "soufle_app_stack"."owner_id", "soufle_app_stack"."is_public_reading", "soufle_app_stack"."is_discussion_stack", "soufle_app_stack"."datetime_canceled" FROM "soufle_app_stack" LEFT OUTER JOIN "soufle_app_stack__readers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__readers"."stack_id" ) LEFT OUTER JOIN "soufle_app_stack__writers" ON ( "soufle_app_stack"."id" = "soufle_app_stack__writers"."stack_id" ) INNER JOIN "soufle_app_follow" ON ( "soufle_app_stack"."id" = "soufle_app_follow"."stack_id" ) WHERE ("soufle_app_stack"."datetime_canceled" IS NULL AND ("soufle_app_stack"."is_public_reading" = True OR "soufle_app_stack__readers"."user_id" = 147 OR "soufle_app_stack__writers"."user_id" = 147 OR "soufle_app_stack"."owner_id" = 147) AND NOT ("soufle_app_stack"."id" IN (SELECT U1."private_stack_id" AS "private_stack_id" FROM "soufle_app_user" U1 WHERE (U1."id" = 147 AND U1."private_stack_id" IS NOT NULL))) AND NOT ("soufle_app_stack"."id" IN (SELECT U1."public_stack_id" AS "public_stack_id" FROM "soufle_app_user" U1 WHERE (U1."id" = 147 AND U1."public_stack_id" IS NOT NULL))) AND "soufle_app_follow"."user_id" = 147) ORDER BY "soufle_app_stack"."datetime_created" DESC
comment:6 by , 10 years ago
Can anyone possibly help with this? I'm stuck on this problem and I don't know how to solve it.
follow-up: 8 comment:7 by , 10 years ago
Can you try your queryset out with django 1.8 please? I've tried to emulate the idea by combining two querysets, one that has distinct applied, and that behaviour is not allowed (in 1.8):
In [9]: Company.objects.filter(name='HI').distinct() & Company.objects.exclude(name='HI') --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-9-b0c9de44f4df> in <module>() ----> 1 Company.objects.filter(name='HI').distinct() & Company.objects.exclude(name='HI') /Users/smeatonj/Development/django/django/db/models/query.py in __and__(self, other) 305 combined = self._clone() 306 combined._merge_known_related_objects(other) --> 307 combined.query.combine(other.query, sql.AND) 308 return combined 309 /Users/smeatonj/Development/django/django/db/models/sql/query.py in combine(self, rhs, connector) 494 "Cannot combine queries once a slice has been taken." 495 assert self.distinct == rhs.distinct, \ --> 496 "Cannot combine a unique query with a non-unique query." 497 assert self.distinct_fields == rhs.distinct_fields, \ 498 "Cannot combine queries with different distinct fields." AssertionError: Cannot combine a unique query with a non-unique query.
So if you hit the same problem as I did in 1.8, then there's little motivation to try to investigate the cause of your original problem in 1.7 since it's explicitly disallowed in 1.8. I would venture that the reason it's disallowed in 1.8 (assuming it was "valid" in 1.7) is for similar reasons that you're seeing now.
The SQL dump provided above doesn't appear to show any distinct clauses, but your original problem description did. We don't have enough information here to figure out the problem.
If you could at least provide the stacktrace when the assert is hit, that might help narrow it down. But, honestly, if this is important enough to your application then you should spend the time needed to create a reproducible series of steps. We help where we can, but I think you'd be hard pressed to find someone willing to try combination after combination trying to arrive at a similar situation that you're in because it'd take you "too much time".
comment:8 by , 10 years ago
Hi Josh,
Thanks for your help.
- Turns out I was wrong about the
.distinct
thing, it's to be applied after the failing code and not before it, so it's not relevant. That was my mistake. In other words.distinct
is definitely not used in the failing query, so we can't expect to get the "Cannot combine a unique query with a non-unique query" error.
- I tried to run my code on Django 1.8, and got the exact same error.
- Here is a traceback of the error: http://i.imgur.com/nM748ne.jpg The frame just above the one visible is the frame in which I do
&
between two querysets. (Didn't include these frames because they have client code.) (This traceback was done on Django 1.8, if you want the same on Django 1.7.7 I can do that too.) If you'll want me to expand any of the "Local vars" sections, let me know.
Also, if you're interested enough and have the time to do a VNC or TeamViewer session to my dev machine, I can show you the whole thing. Of course, I'll totally understand if you're too busy. If so I'll try to do a sample app. (Trying to avoid billing my client for the hours that this will take.)
Thanks,
Ram.
comment:9 by , 10 years ago
It's not likely that just seeing the problem is going to be enough to reproduce on my own. I sympathise with your situation, but you're trying to avoid billing your client by palming the work off for free to open source contributors. We're not tier 3 support. You should (IMO) be billing for bug fixes in situations like this.
I'm happy to meet you half way. If you can put a test case together then we can work on fixing the issue.
Cheers
comment:10 by , 10 years ago
Resolution: | → needsinfo |
---|---|
Status: | new → closed |
by , 9 years ago
Attachment: | and_assertion_error.zip added |
---|
minimal example project exhibiting the error
comment:11 by , 9 years ago
In Django 1.8, with the models
from django.db import models class Year(models.Model): date = models.DateField(auto_now_add=True) class Student(models.Model): year = models.ForeignKey('Year') number = models.PositiveIntegerField() courses = models.ManyToManyField('Course') class Course(models.Model): year = models.ForeignKey('Year') department = models.CharField(max_length=4) number = models.PositiveIntegerField()
the following
course_instance.student_set.all() & year_instance.student_set.exclude(courses__in=[other_course_instance])
will raise
Traceback (most recent call last): File "/home/kevmitch/projects/apprentice/and_assertion_error/enrollment/tests.py", line 24, in test_and result = math101.student_set.all() & year.student_set.exclude(courses__in=[anth100]) File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 211, in __and__ combined.query.combine(other.query, sql.AND) File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 556, in combine w.relabel_aliases(change_map) File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 285, in relabel_aliases child.relabel_aliases(change_map) File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 287, in relabel_aliases self.children[pos] = child.relabeled_clone(change_map) File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/lookups.py", line 185, in relabeled_clone new.rhs = new.rhs.relabeled_clone(relabels) File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 329, in relabeled_clone clone.change_aliases(change_map) File "/home/kevmitch/projects/apprentice/apprentice/env/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 787, in change_aliases assert set(change_map.keys()).intersection(set(change_map.values())) == set() AssertionError
The tests of attached project demonstrates it in action.
Essential elements for reproduction:
- One of the querysets generated from related manager of the ManyToMany target
- The other queryset is generated by exclude()ing on the ManyToManyField from the related manager of a "versioning" ForeignKey model
comment:12 by , 9 years ago
Resolution: | needsinfo |
---|---|
Status: | closed → new |
comment:13 by , 9 years ago
Cc: | added |
---|
comment:14 by , 9 years ago
Severity: | Normal → Release blocker |
---|---|
Triage Stage: | Unreviewed → Accepted |
Bisected to 01f2cf2aecc932d43b20b55fc19a8fa440457b5f
comment:15 by , 9 years ago
Has patch: | set |
---|
Removing the assertion seems to work. Is there a better solution?
comment:16 by , 9 years ago
Patch needs improvement: | set |
---|
comment:19 by , 7 years ago
Resolution: | fixed |
---|---|
Status: | closed → new |
Hi all, unfortunately I have to reopen this issue.
After migrating from Django 1.10.6 to 1.11.9 the error reoccurs:
Code:
qs_out = qs.filter(*out_filters) qs_in = qs.filter(*in_filters) qs_merged = qs_in | qs_out
Vars:
in_filters = [<Q: (NOT (AND: ('contract__customer_id', 1)))>, <Q: (NOT (AND: ('contract__category__deletion', True)))>, <Q: (NOT (AND: ('contract__date_from__lte', datetime.date(2018, 1, 10)), ('contract__category__blocked', True), ('contract__date_until__gte', datetime.date(2018, 1, 10))))>, <Q: (AND: ('id', 0))>]
out_filters = [<Q: (OR: (AND: (NOT (AND: ('contract__source_customer_id', 1))), ('contract__customer_id', 1)), (AND: (NOT (AND: ('contract__source_customer_id', 1))), (NOT (AND: ('contract__customer_id', 1)))))>, <Q: (NOT (AND: ('contract__category__incoming', True)))>]
It seems to fail, if qs_in
or qs_out
is empty. But I'm not sure.
Printing change_map
gives me:
{'T7': u'invoice_customer', u'invoice_customer': 'T7'}
Removing the assert works.
comment:20 by , 7 years ago
Version: | 1.7 → 1.11 |
---|
comment:21 by , 7 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
As the fix for this ticket is released, please open a new ticket with complete details to reproduce. Ideally, you could also bisect to find the commit where the behavior changed.
Couldn't you change the names of the models and values of the data and offer a test case to reproduce the error?