Reverse many to one and forward many to many managers's prefetch object cache should be cleared on add/remove/clear().

[Sharat M R]

class Media(models.Model):
    name = models.CharField(max_length=100)


class Tag(models.Model):
    tag = models.CharField(max_length=100)
    medias = models.ManyToManyField(Media)

    def __unicode__(self):
        return self.tag

Assume tag object below already has three items of Media model in medias which are saved in db. [m1, m2, m3].

tag = Tag.objects.get(id=1)
tag.medias.clear()
print(tag.medias.all())

The above code prints all the three [m1, m2, m3]. Fetching the tag again using get and printing givens an empty list "[ ]"

Failing Testcase

class Test(TestCase):
    def test(self):
        tag = Tag.objects.create(tag='Tag')
        tag.medias.create(name='A')
        tag.medias.create(name='B')
        tag.medias.create(name='C')
        tag = Tag.objects.prefetch_related('medias').get(id=1)
        tag.medias.clear()
        print(tag.medias.all())

I guess its related to prefetch_related function.

[Tim Graham]

I can't reproduce that. Here's my complete test case:

from django.test import TestCase

from .models import Tag


class Test(TestCase):
    def test(self):
        tag = Tag.objects.create(tag='Tag')
        tag.medias.create(name='A')
        tag.medias.create(name='B')
        tag.medias.create(name='C')
        tag = Tag.objects.get(id=1)
        tag.medias.clear()
        print(tag.medias.all())

Also, Django's test suite ​tests the behavior.

Please reopen if you can provide a failing test case.

[Sharat M R]

class Test(TestCase):
    def test(self):
        tag = Tag.objects.create(tag='Tag')
        tag.medias.create(name='A')
        tag.medias.create(name='B')
        tag.medias.create(name='C')
        tag = Tag.objects.prefetch_related('medias').get(id=1)
        tag.medias.clear()
        print(tag.medias.all())

This test print all the three medias for me

[Simon Charette]

From a quick look at the code this looks like a legitimate issue.

The manager classes returned by create_forward_many_to_many_manager and create_reverse_many_to_one_manager factories are both affected.

[Simon Charette]

​PR

LGTM pending some cosmetic changes.

[Tim Graham]

I agree the current behavior might be unexpected, however, I'm not sure if it's a good idea to add this behavior on some manager methods but not others (see #25344)? At least the behavior should be documented, probably.

[Yoong Kang Lim]

I just read #25344, thanks, didn't know there were other methods that were affected.

I think the behaviour should be added in both cases. Is it not reasonable, in general, to expect cache to be cleared on change of data? If the historical cached list is needed for any reason, it could always be stored in a variable.

I do agree that in either case it should be documented. Either that your prefetched values could be out of date, or that any changes via the add(), clear() and remove() methods will clear the cache.

I have updated my patch to cater for add(), remove() in addition to clear(). Happy to amend and add documentation either way we go, but I'll wait for consensus before adding documentation.

[Tim Graham]

Could you make a post on the DevelopersMailingList about this? I want to make sure that there's a consensus that the possible backwards-incompatibilities of clearing existing caches are acceptable.

[Tim Graham]

​django-developers thread

[Tim Graham <timograham@…>]

In d30febb4:

Fixed #26706 -- Made RelatedManager modification methods clear prefetch_related() cache.

[GitHub <noreply@…>]

In 40a2c811:

Refs #26706, Refs #34633 -- Added test for prefetch_related() cache invalidation in ManyRelatedManager.create().