Speed up fixture loading by adding options bulk insert/create

[JorisBenschop]

As per ​this forum discussion, I have created a patch to improve load times for the loaddata command under some circumstances.

Currently the “loaddata” management command uses the obj.save() method for each deserialized object within a fixture. This function first tries an UPDATE statement and, if that fails, tries an INSERT statement. By using the --force_insert a reduction of 50% of queries is achieved.

A second option is to use bulk_create for insertion of multiple records. This improves insertion speed by (n-1/n), or ~99% for insertion of 100 records.

These options are not meant to cover each use case, and therefore are set to optional.

[Simon Charette]

Hello Joris,

This sounds interesting particularly given features like ​test case serialized rollbacks (which are quite slow) are based on top of model serialization. It would have to be a distinct option as bulk_create doesn't fire signals which some setup might require.

Just like any new feature requests though ​they should be discussed on the forum to reach a consensus before being accepted. Given this is a performance related new feature I suggest your proposal come equipped with some details about what kind of improvements users should expect (profiles, benchmarks instead of solely claiming it's fairly inefficient) backed by step to reproduce as well as a PoC that properly deals with other features of serde framework such as natural keys and a plan on how to deal with backends that don't support ignore_conflicts. It might even be a good opportunity to augment ​our performance tracking system with serde benchmarks.

If there is genuine interest in this feature i will develop it and submit. I have this running in my own codebase already.

It that's the case then sharing this code as a standalone package (e.g. django-fast-loaddata) might be a good way to get traction on the above.

Assuming there is interest in moving forward we can then re-open this issue.

[Sarah Boyce]

#35975 was a duplicate

Forum discussion: ​https://forum.djangoproject.com/t/feature-proposal-faster-fixture-loading-via-loaddata-command/36972
PR: ​https://github.com/django/django/pull/18889