Opened 3 months ago

Closed 2 months ago

#35408 closed Cleanup/optimization (fixed)

Optimize post-migrate permission creation

Reported by: Adam Johnson Owned by: Adam Johnson
Component: contrib.auth Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Adam Johnson)

I have often seen django.contrib.auth.management.create_permissions() take a significant amount of time in test run profiles. It can be optimized by batching more of its operations, including making ContentTypeManager.get_for_models() use batch creation.

For a comparison, I profiled 1518 of Django’s tests in modules called “models”:

$ python -m cProfile -o profile runtests.py --parallel 1 *model*

$ python -m pstats profile <<< 'sort cumtime
stats 10000' | less

Before optimization stats:

  • Total 11,938,857 function calls taking 5.349 seconds.
  • 88 calls to create_permissions() take 456ms, ~8.5% of the total time.

After optimization stats:

  • Total 11,359,071 function calls taking 5.035 seconds.
  • 88 calls to create_permissions() now take 239ms, ~4.7% of the toal time.
  • 217ms and 579,786 function calls saved.

Optimization is limited because the post_migrate signal runs once per migrated app config, so there’s no chance to bulk create *all* content types and permissions at once. If we introduced a new “all migrated apps” signal, that could reduce runtime further by batching all creation.

Change History (6)

comment:1 by Natalia Bidart, 3 months ago

Patch needs improvement: set
Triage Stage: UnreviewedAccepted

Accepting following an initial review of the patch which looks sensible. Setting as patch needs improvement due to the comments raised by David and Mariusz.

comment:2 by Adam Johnson, 3 months ago

I repeated the profiling with the latest version of the patch, on top of the latest main commit. The numbers are similar.

Before optimization stats:

  • Total 12,387,798 function calls taking 5.589 seconds.
  • 88 calls to create_permissions() take 483ms, ~8.6% of the total time.

After optimization stats:

  • Total 11,797,519 function calls taking 5.207 seconds.
  • 88 calls to create_permissions() take 241ms, ~4.6% of the total time.
  • 590,279 function calls and 242ms saved.

comment:3 by Adam Johnson, 3 months ago

Patch needs improvement: unset

comment:4 by Sarah Boyce, 2 months ago

Triage Stage: AcceptedReady for checkin

comment:5 by Adam Johnson, 2 months ago

Description: modified (diff)

comment:6 by Sarah Boyce <42296566+sarahboyce@…>, 2 months ago

Resolution: fixed
Status: assignedclosed

In d2c5a30e:

Fixed #35408 -- Optimized post-migrate permission creation.

co-authored-by: Mariusz Felisiak <felisiak.mariusz@…>

Note: See TracTickets for help on using tickets.
Back to Top