QuerySet indexing by __getitem__ gets wrong answer in edge cases
|Reported by:||smoluf||Owned by:||mtredinnick|
|Severity:||Keywords:||queryset index race|
|Cc:||andrew.tennikoff@…, carljm||Triage Stage:||Unreviewed|
|Has patch:||no||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
I've just spent a few hours in pdb tracking this down. I've got about as far as I'm able into the Django libraries looking for the culprit, but I've run out of ideas due to my limited knowledge of Django's inner workings. I need someone with a bit more experience to look at this one.
I have a Membership model which relates Users to Groups.
# models.py class Membership (models.Model): user = models.ForeignKey(User) group = models.ForeignKey(Group) school_year = models.ForeignKey(SchoolYear, related_name='memberships')
This query turns up the following results:
>>> Membership.objects.filter(user__username="jammons") [<Membership: Office of Technology Services (Jeff Ammons, 2008 - 2009)>, <Membership: UPS Fencing Club (Jeff Ammons, 2008 - 2009)>, <Membership: Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)>]
These results are expected and good. However, indexing does the Wrong Thing. For some reason, __getitem__ returns the same model for  and :
>>> qs <Membership: Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)> >>> qs <Membership: UPS Fencing Club (Jeff Ammons, 2008 - 2009)> >>> qs <Membership: Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)> >>>
The result is that the real qs is unreachable, and furthermore, inline formset validation is therefore broken when this occurs, because the wrong self.instance is set on the first form, causing it to .exclude() the wrong instance from its unique_check and resulting in the form believing that it is a duplicate.
Engaging pdb and then entering "continue" (running the query in the debugger without actually inspecting the trace) returns the same answer.
>>> def test(): ... print Membership.objects.filter(user__username="jammons") ... >>> import pdb >>> pdb.runcall(test) > <console>(2)test() (Pdb) continue Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)
HOWEVER: Running pdb and stepping through the code produces the right answer:
>>> pdb.runcall(test) > <console>(2)test() (Pdb) step ...lots of stepping through the various nested calls... (Pdb) step Office of Technology Services (Jeff Ammons, 2008 - 2009)
Upon inspecting django.db.connection.queries, what seems to be happening is that the first two methods hit the _result_cache to do the indexing, while the third runs new queries instead.
Theory 1: This is some sort of bizarre race condition where the _result_cache is being deleted, but somehow lingers on long enough that execution speed is a factor in determining whether the indexing can access it or not.
Theory 2: By trial and error, it seemed that stepping through _clone() (django/db/models/query.py, line 218) in pdb gave the right answer, while allowing the debugger to run it at full speed gave the wrong. Perhaps there's something in __clone() that depends on execution speed. This is not 100%, though, as at least on one occasion it produced the wrong answer.
Theory 3: There is some issue with the laziness of QuerySets and database query time.
Someone else is probably better qualified than I am to speculate on the potential causes. I'm happy to provide more detailed information if you need it. Good luck!
Change History (12)
comment:1 Changed 7 years ago by anonymous
- Component changed from Uncategorized to Core framework
- Needs documentation unset
- Needs tests unset
- Patch needs improvement unset
comment:4 Changed 7 years ago by mtredinnick
- Owner changed from nobody to mtredinnick
- Status changed from new to assigned
Changed 7 years ago by john
comment:8 Changed 7 years ago by jacob
- Resolution set to invalid
- Status changed from assigned to closed
comment:9 Changed 7 years ago by smoluf
- Resolution invalid deleted
- Status changed from closed to reopened
comment:10 follow-up: ↓ 11 Changed 7 years ago by jacob
- Resolution set to wontfix
- Status changed from reopened to closed