my_model.manyrelation_set.all() return empty value

[Floréal Cabanettes]

I have a problem which occurred since django 3.2 (no problem with django 3.1).

I use django rest framework. On the view, I have a model "Run" and a model "Experiment". On the view, I get experiments with :
my_experiments = my_run.experiment_set.all()

But since django 3.2, this returns an empty queryset. If I add a breakpoint in pycharm just after this line, the queryset is no more empty.

Also, this line works perferctly:
my_experiments = Experiment.objects.filter(run_id=my_run.id)

But I think the first code is more beautiful... so why this problem?

[Tim Graham]

Hi, could you please provide minimal models and code to reproduce the issue? Ideally, you could also ​bisect to find the commit where the behavior changed.

[Floréal Cabanettes]

Hi,

I found the commit responsible for this bug:

036f160733 ("Refs #20577 -- Deferred filtering of prefetched related querysets by reverse m2o relation.", 2020-10-06)

The line which cause the problem is on the file django/db/models/fields/related_descriptors.py, line 584 (581 at the commit, 584 on tag 3.2.6) :

queryset._defer_next_filter = True

If I comment this line, the bug disappear.

My code is not public (is for my enterprise, not my choice). I try to describe a similar architecture bellow:

I have 2 models, one is "Run", the other "Experiment", related by a OneToMany relationship:

class Run(model.Model):
    label = models.CharField(max_length=100, null=False, blank=False)
    owner = models.ForeignKey(User, null=True, blank=True, on_delete=models.PROTECT, related_name="run_owner")
    

class Experiment(model.Model):
    run = models.ForeignKey(Run, null=False, on_delete=models.PROTECT)
    index1 = models.ForeignKey(Index, null=False, on_delete=models.PROTECT, related_name="index1")
    index2 = models.ForeignKey(Index, null=True, blank=True, on_delete=models.PROTECT, related_name="index2")

I use the django rest framework to call the server through REST API. For one endpoint, I get a run, and I try to get it's experiments, to then make a copy of them:

@action(detail=True, methods=["post"])
def copy(self, request, pk=None):
    try:
        my_run = self.get_queryset().get(pk=pk)
    except ObjectDoesNotExist:
        return Response({"detail": _("Run with id '%s' does not exists") % pk},
                        status=status.HTTP_404_NOT_FOUND)
    my_experiments = my_run.experiment_set.all()
    for my_experiment in my_experiments:
         my_experiment.pk = None
         ...
         my_experiment.save()

and the problem is that my_experiments is empty with django 3.2.6 (and no more empty with the line commented like said before).

I hope this will help to solve the problem...

[Carlton Gibson]

Hi Floréal — thanks for the further detail.

Can you give some indication of the prefetch usage in the get_queryset() method here please:

my_run = self.get_queryset().get(pk=pk)

Thanks.

[Floréal Cabanettes]

This apply some filters on the data (like filters given in input).

But it's not linked to this problem. I try to replace this line by:

my_run = self.queryset.get(pk=pk)

or by :

my_run = Run.objects.get(pk=pk)

and the problem is still the same.

Replying to Carlton Gibson:

Hi Floréal — thanks for the further detail.

Can you give some indication of the prefetch usage in the get_queryset() method here please:

my_run = self.get_queryset().get(pk=pk)

Thanks.

[Carlton Gibson]

Hi Floréal.

I can't see how to reproduce this with the info given. (There must be something else going on in your project.)

Django's model_fields tests use these models:

class Foo(models.Model):
    a = models.CharField(max_length=10)
    d = models.DecimalField(max_digits=5, decimal_places=3)


def get_foo():
    return Foo.objects.get(id=1).pk


class Bar(models.Model):
    b = models.CharField(max_length=10)
    a = models.ForeignKey(Foo, models.CASCADE, default=get_foo, related_name='bars')

This is equivalent to yours here I think (in the relevant respects).

Adding this test passes:

diff --git a/tests/model_fields/test_foreignkey.py b/tests/model_fields/test_foreignkey.py
index 0cd6d62a55..c51b6d978a 100644
--- a/tests/model_fields/test_foreignkey.py
+++ b/tests/model_fields/test_foreignkey.py
@@ -12,6 +12,14 @@ from .models import Bar, FkToChar, Foo, PrimaryKeyCharModel
 
 class ForeignKeyTests(TestCase):
 
+    def test_reverse_relation(self):
+        a = Foo.objects.create(id=1, a='abc', d=Decimal('12.34'))
+        # Bar FK defaults to Foo with id=1
+        b = Bar.objects.create(b='bcd')
+        c = Bar.objects.create(b='bcd')
+        d = Bar.objects.create(b='bcd')
+        self.assertSequenceEqual(a.bars.all(), [b, c, d])
+
     def test_callable_default(self):
         """A lazy callable may be used for ForeignKey.default."""
         a = Foo.objects.create(id=1, a='abc', d=Decimal('12.34'))

This is already covered elsewhere in the test suite (many times).

Can I ask you to provide a minimal reproduce of your issue?
Thanks.