Make Model.refresh_from_db() clear prefetch related caches

[Ming Qin]

The method currently starts with underscore, meaning it's not supposed to be accessed from the outside. But I find that sometimes I have to use this method to clear the cache, otherwise the outdated data will be returned. The following test shows a scenario where this method is useful.

    def setUpTestData(cls):
        cls.book1 = Book.objects.create(title='Les confessions Volume I')
        cls.book2 = Book.objects.create(title='Candide')
        cls.author1 = AuthorWithAge.objects.create(name='Rousseau', first_book=cls.book1, age=70)
        cls.author2 = AuthorWithAge.objects.create(name='Voltaire', first_book=cls.book2, age=65)
        cls.book1.authors.add(cls.author1)
        cls.book2.authors.add(cls.author2)
        FavoriteAuthors.objects.create(author=cls.author1, likes_author=cls.author2)

    def test_remove_prefetched_objects(self):
        """
        Test RelatedManager._remove_prefetched_objects method
        """
        books = Book.objects.prefetch_related('first_time_authors')
        book1 = books.get(title='Les confessions Volume I')
        book2 = books.get(title='Candide')

        # Update the related object - author1. So book1 has no first-time author now, but still has one in cache
        author1 = book1.authors.first()
        author1.first_book = book2
        author1.save()

        self.assertQuerysetEqual(
            book1.first_time_authors.all(), ['<Author: Rousseau>'], msg="Expect the outdated cached data")

        # Now clear the prefetched objects
        book1.first_time_authors._remove_prefetched_objects()

        self.assertQuerysetEqual(
            book1.first_time_authors.all(), [], msg="Expect the fresh data from database")

I'm working on opening a PR to add the above test case, and create an alias of that method with no underscore. Let me know what you think.

[Carlton Gibson]

Not sure about the value of adding this. We already have ​`refresh_from_db()` available.

A public remove_prefetched_objects() would be more fine-grained, yes, but I'm not sure it would be worth the extra API surface area, documentation and potential for user-confusion.
("Just use refresh_from_db()" is nice and clear.)

[Tim Graham]

I don't really like an additional method either but refresh_from_db() only reloads model fields -- it doesn't clear related manager caches, so I guess this might be okay. Prefetch related cache clearing was added in #26706.

[Ming Qin]

Yeah as Tim said, refresh_from_db() doesn't help in this case. I did an experiment just to confirm:

>>> book1.refresh_from_db()
>>> book1.refresh_from_db(fields=["authors"])
>>> list(book1.authors.all())  # still hits the prefetched cache

[Simon Charette]

Could you confirm refresh_from_db(fields=['authors']) works on Django 2.1? If not this might be an omission in #29076.

[Ming Qin]

Replying to Simon Charette:

Could you confirm refresh_from_db(fields=['authors']) works on Django 2.1? If not this might be an omission in #29076.

No, it doesn't work. Still hits the cache. Tested on master, 2.1rc1, and 2.1.
I'll be happy to fix refresh_from_db, instead of opening this new API.

Does the following behavior sound good?

1. If refresh_from_db() is called without argument, it won't clear prefetched cache.
2. If it's called specifically with an related field, and that field is many-to-many or many-to-one field, the prefetched cache will be cleared.

[Simon Charette]

Replying to Ming Qin:

No, it doesn't work. Still hits the cache. Tested on master, 2.1rc1, and 2.1.
I'll be happy to fix refresh_from_db, instead of opening this new API.

Does the following behavior sound good?

1. If refresh_from_db() is called without argument, it won't clear prefetched cache.
2. If it's called specifically with an related field, and that field is many-to-many or many-to-one field, the prefetched cache will be cleared.

1. No, it should clear all fields prefetched cache like it does right now with non-prefetched related manager cache.
2. I haven't investigated but it looks like ManyToManyField.is_cached and ManyToMany.delete_cached_value might need to be redefined from Field in order to take instance._prefetched_objects_cache into consideration as well. It'd be interesting to test how prefetched foreign keys and o2o behave here as well.

[Ming Qin]

​PR

[Tim Graham <timograham@…>]

In cfb4845:

Fixed #29625 -- Made Model.refresh_from_db() clear prefetch related caches.