model_to_dict() makes separate db calls for each missing field

[Chris Hranj]

Currently, if a model_to_dict is called on a model instance that does not already contain all of its fields (as a result of a .only(), etc), it will make an individual database call for every single missing field when attempting to access it, unless the fields argument is passed and happens to contain only fields from the .only() used to query the object originally.

The following example explains it more clearly (I'm using shell_plus (​https://django-extensions.readthedocs.io/en/latest/shell_plus.html) to see the database interactions):

>>> from django.forms.models import model_to_dict
>>> sample_book = Book(
...     title='Some Book',
...     author='Jane Doe',
...     description='A good book.',
...     publisher='ABC Publishing',
...     some_field='some text',
...     some_other_field='some more text',
... )
>>> sample_book.save()
INSERT INTO "books_book" ("title", "author", "description", "publisher", "some_field", "some_other_field")
VALUES ('Some Book', 'Jane Doe', 'A good book.', 'ABC Publishing', 'some text', 'some more text')

Execution time: 0.001141s [Database: default]

>>> book = Book.objects.only('title').first()
SELECT "books_book"."id",
       "books_book"."title"
  FROM "books_book"
 ORDER BY "books_book"."id" ASC
 LIMIT 1

Execution time: 0.000123s [Database: default]

>>> model_to_dict(book)
SELECT "books_book"."id",
       "books_book"."author"
  FROM "books_book"
 WHERE "books_book"."id" = 3

Execution time: 0.000098s [Database: default]

SELECT "books_book"."id",
       "books_book"."description"
  FROM "books_book"
 WHERE "books_book"."id" = 3

Execution time: 0.000149s [Database: default]

SELECT "books_book"."id",
       "books_book"."publisher"
  FROM "books_book"
 WHERE "books_book"."id" = 3

Execution time: 0.000067s [Database: default]

SELECT "books_book"."id",
       "books_book"."some_field"
  FROM "books_book"
 WHERE "books_book"."id" = 3

Execution time: 0.000064s [Database: default]

SELECT "books_book"."id",
       "books_book"."some_other_field"
  FROM "books_book"
 WHERE "books_book"."id" = 3

Execution time: 0.000062s [Database: default]

Ideally, this should make fewer (or just one) database calls if there are missing fields.

Please let me know if this would be a welcome/worthwhile optimization. I would love to work on it, as I've never contributed to django before and I would really like to. Thank you!

[Carlton Gibson]

Hi Chris.

Thanks for the report. (Unless you have an unforeseen concrete proposal in mind) I think we have to say wontfix here.

model_to_dict is just using the field properties. It has no knowledge of whether they deferred or not. (And that's not something we'd add there.)

So, you really need to be passing in objects that already have the required fields loaded. That you're not doing this suggests you're over-using `only() here. See again the warning note:

The defer() method (and its cousin, only(), below) are only for advanced use-cases. They provide an optimization for when you have analyzed your queries closely and understand exactly what information you need and have measured that the difference between returning the fields you need and the full set of fields for the model will be significant.

If you're sure you're using only() correctly, your best bet is probably to add a defer(None) and re-fetch once before running through model_to_dict().