Speed up fixture loading by bulk insert

[JorisBenschop]

I noticed that loading many fixtures is fairly inefficient, in that it tries an update and then an insert for each record. I would propose an additional option to the command line (—bulk-insert) that only runs the insert queries, but also detects if subsequent items in the fixtures are from the same model, so they may be inserted using bulk_insert. If there is genuine interest in this feature i will develop it and submit. I have this running in my own codebase already.
Please also indicate if there is no interest, and i spend my energy elsewhere =)

[Simon Charette]

Hello Joris,

This sounds interesting particularly given features like ​test case serialized rollbacks (which are quite slow) are based on top of model serialization. It would have to be a distinct option as bulk_create doesn't fire signals which some setup might require.

Just like any new feature requests though ​they should be discussed on the forum to reach a consensus before being accepted. Given this is a performance related new feature I suggest your proposal come equipped with some details about what kind of improvements users should expect (profiles, benchmarks instead of solely claiming it's fairly inefficient) backed by step to reproduce as well as a PoC that properly deals with other features of serde framework such as natural keys and a plan on how to deal with backends that don't support ignore_conflicts. It might even be a good opportunity to augment ​our performance tracking system with serde benchmarks.

If there is genuine interest in this feature i will develop it and submit. I have this running in my own codebase already.

It that's the case then sharing this code as a standalone package (e.g. django-fast-loaddata) might be a good way to get traction on the above.

Assuming there is interest in moving forward we can then re-open this issue.