QuerySet.in_bulk() drops annotations selected by values() and values_list().

[Yassin Bahri]

QuerySet.in_bulk() silently drops annotations selected by .values() or .values_list() when the field used as the dictionary key isn't included in the selected fields.

This is reproducible on current main at 5babd2e21ac0877d66c5452da17b97608b337fba and on stable/6.1.x at 4b0185a5fb, using SQLite.

Given these models:

class Author(models.Model):
    name = models.CharField(max_length=100)


class Article(models.Model):
    headline = models.CharField(max_length=100)
    author = models.ForeignKey(Author, models.CASCADE)

The following reproduces the issue:

from django.db.models import F

author = Author.objects.create(name="Author 1")
article = Article.objects.create(
    headline="Article 1",
    author=author,
)

result = (
    Article.objects.annotate(author_name=F("author__name"))
    .values("headline", "author_name")
    .in_bulk([article.pk])
)

print(result)

The actual result omits the selected annotation:

{
    article.pk: {
        "headline": "Article 1",
    }
}

The expected result is:

{
    article.pk: {
        "headline": "Article 1",
        "author_name": "Author 1",
    }
}

The same problem occurs with .values_list():

result = (
    Article.objects.annotate(author_name=F("author__name"))
    .values_list("headline", "author_name")
    .in_bulk([article.pk])
)

The actual value is:

{article.pk: ("Article 1",)}

instead of:

{article.pk: ("Article 1", "Author 1")}

Support for chaining in_bulk() after values() and values_list() was added in #36605 by ​1820d35.

When in_bulk() internally adds the missing key field to the projection, it rebuilds the projection from query.values_select. Selected annotations are stored separately and are therefore omitted from the rebuilt projection.

[Jacob Walls]

Let's add back the test for extra() now that there is special handling for it.

[Jacob Walls <jacobtylerwalls@…>]

In f14e293:

Fixed #37312, Refs #36605 -- Fixed annotation preservation and key selection in in_bulk().

Bug in 1820d35b17f0a95f4ce888971b9ca0c7a3697c83.

Thank you to Jacob Walls for the review.

[Jacob Walls <jacobtylerwalls@…>]

In ef3fc805:

[6.1.x] Fixed #37312, Refs #36605 -- Fixed annotation preservation and key selection in in_bulk().

Bug in 1820d35b17f0a95f4ce888971b9ca0c7a3697c83.

Thank you to Jacob Walls for the review.

Backport of f14e293cd7444e064b9c9ba471fd5aa7e6132ea6 from main.