Allow for fast path model loading
|Reported by:||akaariai||Owned by:||akaariai|
|Component:||Database layer (models, ORM)||Version:||master|
|Cc:||Triage Stage:||Ready for checkin|
|Has patch:||yes||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
I have been playing around with the idea of pushing the model construction to a class method of the model. This will allow very fast reads from database, around 2x faster for normal model, 4x faster for deferred model when you override the from_db() so that it constructs the model directly using the model.dict and doesn't send any signals. With signal sending the speed different is 1.2x for normal model, ~3x for deferred model.
Of course, you can't do that by default. This breaks multiple things: setattr not called (and descriptors don't work either), overridden init not called and of course no signals sent. So, the default would need to be old good __init__ way, or we would need to make a really big backwards compatibility break.
Still, in most projects where model init is a performance bottleneck it will be for just one or two models. 90%+ of models do not need the signals, init call nor do they have any setter of interest in the from DB case. For these models it would be trivial to create a "fast path readonly proxy" for the performance bottleneck cases.
So, by adding the from_db method we could allow for really fast init for those who need it. The cost is around 5% slower for the default __init__ case.
Actually, if you are not going to write the data back to the DB you can throw away the model._state, too. Then you get this kind of results:
Running 'model_init_signals' benchmark ... Min: 0.007368 -> 0.002830: 2.6035x faster Avg: 0.007430 -> 0.002847: 2.6100x faster Significant (t=347.070697) Stddev: 0.00006 -> 0.00001: 5.6553x smaller (N = 18) Running 'query_all' benchmark ... Min: 0.006640 -> 0.002699: 2.4600x faster Avg: 0.006684 -> 0.002713: 2.4636x faster Significant (t=687.137094) Stddev: 0.00002 -> 0.00001: 2.3581x smaller (N = 18) Running 'query_defer' benchmark ... Min: 0.016939 -> 0.003011: 5.6257x faster Avg: 0.017019 -> 0.003028: 5.6211x faster Significant (t=1300.019953) Stddev: 0.00004 -> 0.00001: 3.9626x smaller (N = 18)
I think the 5% slowdown for the default __init__ path is worth it so that those who really need speed can achieve the above shown performance.
A really barebones patch at: https://github.com/akaariai/django/compare/fast_init - this can be used to run benchmarks but can't be included in Django as is - it doesn't have the safe __init__ as the default.
Change History (12)
comment:1 Changed 19 months ago by apollo13
- Needs documentation unset
- Needs tests unset
- Patch needs improvement unset
- Triage Stage changed from Unreviewed to Accepted
comment:3 Changed 19 months ago by akaariai
- Owner changed from nobody to akaariai
- Patch needs improvement set