QuerySet indexing by __getitem__ gets wrong answer in edge cases
|Reported by:||Stefan Moluf||Owned by:||Malcolm Tredinnick|
|Severity:||Keywords:||queryset index race|
|Cc:||andrew.tennikoff@…, Carl Meyer||Triage Stage:||Unreviewed|
|Has patch:||no||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
I've just spent a few hours in pdb tracking this down. I've got about as far as I'm able into the Django libraries looking for the culprit, but I've run out of ideas due to my limited knowledge of Django's inner workings. I need someone with a bit more experience to look at this one.
I have a Membership model which relates Users to Groups.
# models.py class Membership (models.Model): user = models.ForeignKey(User) group = models.ForeignKey(Group) school_year = models.ForeignKey(SchoolYear, related_name='memberships')
This query turns up the following results:
>>> Membership.objects.filter(user__username="jammons") [<Membership: Office of Technology Services (Jeff Ammons, 2008 - 2009)>, <Membership: UPS Fencing Club (Jeff Ammons, 2008 - 2009)>, <Membership: Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)>]
These results are expected and good. However, indexing does the Wrong Thing. For some reason,
__getitem__ returns the same model for
>>> qs <Membership: Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)> >>> qs <Membership: UPS Fencing Club (Jeff Ammons, 2008 - 2009)> >>> qs <Membership: Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)> >>>
The result is that the real
qs is unreachable, and furthermore, inline formset validation is therefore broken when this occurs, because the wrong self.instance is set on the first form, causing it to .exclude() the wrong instance from its unique_check and resulting in the form believing that it is a duplicate.
Engaging pdb and then entering "continue" (running the query in the debugger without actually inspecting the trace) returns the same answer.
>>> def test(): ... print Membership.objects.filter(user__username="jammons") ... >>> import pdb >>> pdb.runcall(test) > <console>(2)test() (Pdb) continue Adelphian Concert Choir (Jeff Ammons, 2008 - 2009)
HOWEVER: Running pdb and stepping through the code produces the right answer:
>>> pdb.runcall(test) > <console>(2)test() (Pdb) step ...lots of stepping through the various nested calls... (Pdb) step Office of Technology Services (Jeff Ammons, 2008 - 2009)
django.db.connection.queries, what seems to be happening is that the first two methods hit the _result_cache to do the indexing, while the third runs new queries instead.
Theory 1: This is some sort of bizarre race condition where the _result_cache is being deleted, but somehow lingers on long enough that execution speed is a factor in determining whether the indexing can access it or not.
Theory 2: By trial and error, it seemed that stepping through _clone()
(django/db/models/query.py, line 218) in pdb gave the right answer, while allowing the debugger to run it at full speed gave the wrong. Perhaps there's something in
__clone() that depends on execution speed. This is not 100%, though, as at least on one occasion it produced the wrong answer.
Theory 3: There is some issue with the laziness of QuerySets and database query time.
Someone else is probably better qualified than I am to speculate on the potential causes. I'm happy to provide more detailed information if you need it. Good luck!
Change History (12)
comment:4 Changed 8 years ago by
|Owner:||changed from nobody to Malcolm Tredinnick|
|Status:||new → assigned|