Opened 7 months ago
Closed 6 months ago
#35408 closed Cleanup/optimization (fixed)
Optimize post-migrate permission creation
Reported by: | Adam Johnson | Owned by: | Adam Johnson |
---|---|---|---|
Component: | contrib.auth | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Ready for checkin | |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description (last modified by )
I have often seen django.contrib.auth.management.create_permissions()
take a significant amount of time in test run profiles. It can be optimized by batching more of its operations, including making ContentTypeManager.get_for_models()
use batch creation.
For a comparison, I profiled 1518 of Django’s tests in modules called “models”:
$ python -m cProfile -o profile runtests.py --parallel 1 *model* $ python -m pstats profile <<< 'sort cumtime stats 10000' | less
Before optimization stats:
- Total 11,938,857 function calls taking 5.349 seconds.
- 88 calls to
create_permissions()
take 456ms, ~8.5% of the total time.
After optimization stats:
- Total 11,359,071 function calls taking 5.035 seconds.
- 88 calls to
create_permissions()
now take 239ms, ~4.7% of the toal time. - 217ms and 579,786 function calls saved.
Optimization is limited because the post_migrate
signal runs once per migrated app config, so there’s no chance to bulk create *all* content types and permissions at once. If we introduced a new “all migrated apps” signal, that could reduce runtime further by batching all creation.
Change History (6)
comment:1 by , 7 months ago
Patch needs improvement: | set |
---|---|
Triage Stage: | Unreviewed → Accepted |
comment:2 by , 7 months ago
I repeated the profiling with the latest version of the patch, on top of the latest main
commit. The numbers are similar.
Before optimization stats:
- Total 12,387,798 function calls taking 5.589 seconds.
- 88 calls to create_permissions() take 483ms, ~8.6% of the total time.
After optimization stats:
- Total 11,797,519 function calls taking 5.207 seconds.
- 88 calls to create_permissions() take 241ms, ~4.6% of the total time.
- 590,279 function calls and 242ms saved.
comment:3 by , 7 months ago
Patch needs improvement: | unset |
---|
comment:4 by , 6 months ago
Triage Stage: | Accepted → Ready for checkin |
---|
comment:5 by , 6 months ago
Description: | modified (diff) |
---|
Accepting following an initial review of the patch which looks sensible. Setting as patch needs improvement due to the comments raised by David and Mariusz.