Opened 5 weeks ago

Closed 12 days ago

#36088 closed Cleanup/optimization (fixed)

Avoid unnecessary DEFAULT usage on bulk_create for models with db_default fields

Reported by: Simon Charette Owned by: Simon Charette
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: unnest insert db_default default bulk_create
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Simon Charette)

When bulk_create is used for models composed of fields with db_default the resulting INSERT statement use DEFAULT placeholders to signify that a field must use the database defined default.

For example, the following code

class Author(models):
    name = models.CharField(max_length=100)
    created_at = models.DateTimeField(db_default=Now())

Author.objects.bulk_create([Author(name="foo"), Author(name="bar")])

Will result in the following SQL

INSERT INTO author (name, created_at) VALUES (%s, DEFAULT), (%s, DEFAULT)

But in cases where no db_default is provided for all bulk-created instances there is no point in specifying DEFAULT for each row as that's what the database will do if the column is not specified at all. In other words the above SQL is equivalent to

INSERT INTO author (name) VALUES (%s), (%s)

but the latter query simplification provide benefits:

Firstly, it would allow the UNNEST optimization introduced in #35936 (a16eedcf9c69d8a11d94cac1811018c5b996d491) to be enabled for models that define db_default fields. Alas since DEFAULT is an expression and the optimization must be disabled in their presence no models making use of db_default can take advantage of it.

In other words, on Postgres, the SQL could be

INSERT INTO author (name) SELECT * FROM unnest([%s, %s])

which has demonstrated benefits.

Secondly, pruning the field would avoid having to provide the db_default expression for all model instances on backends that don't support DEFAULT in bulk-inserts such as Oracle and SQLite. In other words the following SQL would be avoided

INSERT INTO author (name, created_at) VALUES (%s, NOW()), (%s, NOW())

Lastly, it just make the query smaller as no DEFAULT has to be provided for each row for each columns with a defined db_default which surely reduce the parsing time on the backend.

Change History (5)

comment:1 by Simon Charette, 5 weeks ago

Summary: Avoided unnecessary DEFAULT usage on bulk_create.Avoid unnecessary DEFAULT usage on bulk_create.

comment:2 by Simon Charette, 5 weeks ago

Description: modified (diff)
Summary: Avoid unnecessary DEFAULT usage on bulk_create.Avoid unnecessary DEFAULT usage on bulk_create for models with db_default fields

comment:3 by Mariusz Felisiak, 5 weeks ago

Triage Stage: UnreviewedAccepted

comment:4 by Mariusz Felisiak, 12 days ago

Triage Stage: AcceptedReady for checkin

comment:5 by Mariusz Felisiak <felisiak.mariusz@…>, 12 days ago

Resolution: fixed
Status: assignedclosed

In 4608d34b:

Fixed #36088 -- Avoided unnecessary DEFAULT usage on bulk_create().

When all values of a field with a db_default are DatabaseDefault, which
is the case most of the time, there is no point in specifying explicit
DEFAULT for all INSERT VALUES as that's what the database will do anyway
if not specified.

In the case of PostgreSQL doing so can even be harmful as it prevents
the usage of the UNNEST strategy and in the case of Oracle, which
doesn't support the usage of the DEFAULT keyword, it unnecessarily
requires providing literal db defaults.

Thanks Lily Foote for the review.

Note: See TracTickets for help on using tickets.
Back to Top