Accessing a related field after using defer() or only() unexpectedly overwrites unsaved state on model instances

[Erick]

When working with a model instance that was originally loaded with only() or defer(), accessing any related object not cached on the instance reverts unrelated fields to the value currently present in the database.

This toy example reproduces the behavior:

class SomethingElse(Model):
    ...

class Something(Model):
    foo = CharField(max_length=2)
    bar = CharField(max_length=2)
    other = ForeignKey(SomethingElse)


Something.objects.create(
    foo="ab",
    bar="cd",
    other=SomethingElse.objects.create()
)

something = Something.objects.only("bar").first()
something.foo = "ef"
assert something.foo == "ef"

something.other   # Accessing this deferred related field triggers the behavior
assert something.foo == "ef"  # AssertionError

This is surprising behavior, because application code may interact with model instances without knowing how they were loaded from the database.

[Simon Charette]

I cannot reproduce against main or 5.2 (which you reported the issue against) with

tests/defer/tests.py

@@ -184 +184 @@
         with self.assertNumQueries(1):
             self.assertEqual(primary.related_id, self.p1.related_id)
 
+    def test_related_fetching(self):
+        Primary.objects.create(
+            name="foo",
+            value="bar",
+            related=Secondary.objects.create(first="first", second="second"),
+        )
+        primary = Primary.objects.only("name").first()
+        primary.value = "ef"
+        self.assertEqual(primary.value, "ef")
+        primary.related
+        self.assertEqual(primary.value, "ef")
+
 
 class BigChildDeferTests(AssertionMixin, TestCase):
     @classmethod

At first I thought you ran into #35950 (which was fixed in 5.1.4) but I can't reproduce against 5.1.3 either.

I'm afraid you might have omitted some details here.

[Erick]

You're right, I over-simplified it. This appears to happen when:

1. An attribute is set on the model instance, and the name of the impacted field ("value" in your example) is a substring of that attribute name;
2. The impacted field is accessed in the model instance constructor;

This repros on main:

diff --git a/tests/defer/models.py b/tests/defer/models.py
index 560e54c8c0..5b5f8fb86a 100644
--- a/tests/defer/models.py
+++ b/tests/defer/models.py
@@ -18,6 +18,9 @@ class Primary(models.Model):
     def __str__(self):
         return self.name
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.value
 
 class PrimaryOneToOne(models.Model):
     name = models.CharField(max_length=50)
diff --git a/tests/defer/tests.py b/tests/defer/tests.py
index 989b5c63d7..11b7042a2f 100644
--- a/tests/defer/tests.py
+++ b/tests/defer/tests.py
@@ -360,3 +360,17 @@ class DeferredRelationTests(TestCase):
         obj.second  # Accessing a deferred field.
         with self.assertNumQueries(0):
             obj.primary_o2o
+
+
+    def test_related_fetching(self):
+        Primary.objects.create(
+            name="foo",
+            value="bar",
+            related=Secondary.objects.create(first="first", second="second"),
+        )
+        primary = Primary.objects.only("name").first()
+        setattr(primary, "_any_value",  "what")
+        primary.value = "ef"
+        self.assertEqual(primary.value, "ef")
+        primary.related
+        self.assertEqual(primary.value, "ef")
\ No newline at end of file

So it is a rather niche condition, admittedly. But still surprising.

[Simon Charette]

Accessing deferred fields in __init__ is going to cause you a lot of trouble #22858, #31435, #32660 and ​is documented to be problematic.

It causes a silent N+1 query which explains why your value is overriden when value is not part of the inclusion mask (accessing self.value will cause it to be retrieved it from the database and then assigned on the model instance).

If you really need to implement field retrieval in __init__ you should ​gate these retrieval behind get_deferred_fields.

def __init__(self, *args, **kwargs):
    super().__init__(self, *args, **kwargs)
    if "value" not in self.get_deferred_fields():
        self.value

[Erick]

That makes sense, appreciate the explanation. I wonder if it would be reasonable to update the docs to mention this bad behavior; as written, I would have passed off the admonition about __init__ as probably not related given that it just talks about infinite recursion.

Additionally, referring to model fields within __init__ may potentially result in infinite recursion errors in some circumstances.

Happy to contribute a PR if you think it's a good idea.