prefetch_related child queryset does not update on create

[Lucas Wiman]

When a child foreign key relationship has been prefetched, calling
the .create method on the queryset does not update the queryset.
I've reproduced this bug in Django 1.5.4 and Django 1.6.

How to reproduce:

models.py

from django.db import models
 
class Parent(models.Model):
    pass
 
 
class Child(models.Model):
    parent = models.ForeignKey(Parent)

In the shell:

>>> p = Parent.objects.create()
>>> list(p.child_set.all())
[]
>>> child = p.child_set.create()
>>> list(p.child_set.all())
[<Child: Child object>]
>>> 
>>> p2 = Parent.objects.create()
>>> parents = Parent.objects.filter(pk=p2.id).prefetch_related('child_set')
>>> [p2_prefetched] = parents
>>> list(p2_prefetched.child_set.all())
[]
>>> p2_prefetched.child_set.create()
<Child: Child object>
>>> list(p2_prefetched.child_set.all())
[]

The last expression should return a list with one child in it, but returns an empty list instead.

[Luke Plant]

This is not a bug, I'm afraid. The results of DB queries that have already been run are never updated automatically by the Django ORM, and that is by design. (There may be some small exceptions, such as when you update a FK object, and the FK ID on that object may be updated as well, but only those that can be done with no dependency tracking). When you specify 'prefetch_related', you are specifying this exact behaviour i.e. the 'child_set.all()' is not lazy, but prefetched in a single query.

Changing this would really require an identity mapper, and a very fundamental change to the way the Django ORM works.

[Lucas Wiman]

This is not a bug, I'm afraid.

It's very unintuitive that the child_set won't keep track of what elements are in it when it is mutated by its own methods. This seems to break the "iterable" abstraction.

There may be some small exceptions, such as when you update a FK object, and the FK ID on that object may be updated as well, but only those that can be done with no dependency tracking

Isn't that precisely the behavior that's being described here? The foreign key relationship should know that a new entry was created, because the create method was called on the child_set itself. Create should invalidate the cache, because there's no way the cache can remain valid after create is called. Couldn't create set self._results_cache to None, and self.prefetch_done to False when create is called?

[Luke Plant]

Sorry, that's just how the ORM works. Objects that represent collections do not keep track of their elements, because they represent querysets i.e. queries that may or may not yet have been evaluated, not actual collections of objects. If you have:

    my_objects = Foo.objects.all().filter(bar=1)
    list(my_objects) # evaluate query
    my_objects.update(bar=2)

then you will find that the 'update' has not affected anything in my_objects - either by changing the instances, or by removing them from the collection (since they no longer match the requirement bar=1).

In the same way, p.child_set does not keep track of elements that are referred to. When you call all(), it executes a query every time, (rather than tracking creates/adds/deletes etc.). If you have used prefetch_related, however, it never executes a query when you just do all() because it has been prefetched. This is exactly what prefetch_related is supposed to do - the all() will not return data to reflect what is in the DB at that moment in time, but what was in the DB when the query was first evaluated.

[Nick Retallack]

But how do I bust the cache?