Performance regression with end to end migrations on v5.x

[William Yardley]

We've recently updated from Django 4.2 to 5.2. We have a nightly job that runs all our migrations end to end.
Even after squashing _all_ our migrations (basically declaring migration bankruptcy fully) recently, there are ~ 800 migrations across ~ 95 apps.

The nightly run doubled in time from 45m to 90m just from this update. Profiling a from-scratch migration shows about 93% of the time in ProjectState rendering and almost none in cursor.execute.

Some quick Claude analysis (so the usual caveats apply) suggests this could relate to changes with caching of swappable-settings lookups, and _possibly_ also how field choices get normalized?

I know there have been some (long) past discussions mentioning that Django isn't trying to optimize this path too much. But to me, it does seem like a pretty significant regression for this to take twice as long.

Happy to provide other details if it's helpful; unfortunately, providing a full clean reproduction may be difficult, since this is happening in a fairly large internal codebase.

Possibly related: #34687 and #28033

[Jacob Walls]

Thanks for the report. Some additional information will be needed to move this forward, I'm afraid. If you have a suspicion choices are involved, then scripting out a test project with a dozen migrations that only tinker with choices, and measuring if it's slower on 5.2 would help.

[William Yardley]

Hi - thanks. Would appreciate the opportunity to provide input / examples vs. just having the ticket closed. I can put up a very basic skeleton project that should help demonstrate the issue on a smaller scale, or at least be testable independently.

​https://github.com/wyardley/django-42-52-migrations

Does this (AI generated) repro repo work as a starting point to testing / reproducing the issue?

[Sarah Boyce]

Thank you for the test project and analysis! Performance regression mostly due to 500e01073adda32d5149624ee9a5cb7aa3d3583f (Refs #31262)

On my machine, between 4.2 and 5.0 the migrations take almost 3x more time. The performance hasn't changed in such a significant way between 5.0 and 6.1 but I have seen this improve significantly on main, though still about x2 than 4.2. Would be interesting to see if others experience the same?

Within the "why" of the test project README:

Field.choices became a property that runs normalize_choices on every assignment. In 4.2 it was a plain attribute. Rebuilding state clones every field of every affected model on each operation, so the same choice lists get normalized repeatedly.

This refers to this commit 500e01073adda32d5149624ee9a5cb7aa3d3583f (#31262)
We also saw this caused a performance regression visible on django-asv: ​https://django.github.io/django-asv/#form_benchmarks.select_date_widget.benchmark.DateWidget.time_selectdatewidget?machine=ubuntu-latest&python=3.10&commits=68a8996b-9a9620dd
I also saw that python bench.py large take 60% more time after 500e01073adda32d5149624ee9a5cb7aa3d3583f

A rough solution for the above could be:

django/utils/choices.py

@@ -69 +69 @@
             yield value_or_group, label_or_nested
 
 
+class NormalizedChoiceList(list):
+    """Concrete choices already in canonical form."""
+
+
 def normalize_choices(value, *, depth=0):
     """Normalize choices values consistently for fields and widgets."""
     # Avoid circular import when importing django.forms.
     from django.db.models.enums import ChoicesType
 
     match value:
+        case NormalizedChoiceList():
+            # Avoid renormalizing already normalized choices.
+            return value
         case BaseChoiceIterator() | Promise() | bytes() | str():
             # Avoid prematurely normalizing iterators that should be lazy.
             # Because string-like types are iterable, return early to avoid
@@ -107 +114 @@
 
     try:
         # Recursive call to convert any nested values to a list of 2-tuples.
-        return [(k, normalize_choices(v, depth=depth + 1)) for k, v in value]
+        return NormalizedChoiceList(
+            (k, normalize_choices(v, depth=depth + 1)) for k, v in value
+        )
     except (TypeError, ValueError):
         # Return original value for the system check to raise if it has items
         # that are not iterable or not 2-tuples:

tests/utils_tests/test_choices.py

@@ -395 +395 @@
                     normalize_choices((lambda: (yield from value))()),
                     value,
                 )
+
+    def test_normalized_choices_does_not_renormalize(self):
+        choices = normalize_choices({"A": "Alpha"})
+        self.assertEqual(choices, [("A", "Alpha")])
+        self.assertIs(normalize_choices(choices), choices)

This resulted in a speed up to be similar to 4.2 levels. But this needs a lot more investigation and testing

Performance improvement is not my strong point so I would appreciate further input here

---

As a side note, tickets are closed when they are waiting for further input from authors but are reopened when new information is given.
A closed state isn't final but allows us to avoid reminding individuals periodically to provide more information and then closing after an arbitrary amount of time if they don't get back to us.

[Akshat Sparsh]

Thanks for the pointer, Sarah. I reproduced this on current main and looked more closely at the normalized-list approach.

Using the public large benchmark on Python 3.13.12, current main (957d0cee71) had a median of 7.861 s and my local draft had a median of 4.318 s across 12 randomized pairs. The draft was faster in all 12 pairs. On Python 3.12.13 it was still 13.3% slower than Django 4.2.20, so this seems to recover most, but not all, of the regression.

I found one issue with returning a marked list unchanged: choices are mutable. If an already-normalized list or one of its nested groups is changed, the next assignment still needs to normalize the new value.

My current draft shares a dirty flag across each normalized choices tree. Ordinary list mutations mark the tree dirty, and the next call rebuilds it. Copying and pickling produce regular lists. The focused tests pass on Python 3.12, 3.13, and 3.14. A fresh full serial run completed successfully: 19,015 tests, with 1,676 skips and 4 expected failures.

One compatibility concern remains. The result is still a list subclass, so isinstance(choices, list) works, but exact-type consumers can notice the difference. PyYAML's safe_dump() is one example unless the value is converted to a regular list first.

Would you prefer that I continue with the mutation-aware list approach, or look for a narrower optimization in the field or migration cloning path?

AI disclosure: I used GPT 5.6 Sol to help trace the call path, draft the local implementation and tests, and orchestrate the local verification above. All timings quoted here are from those local runs.

[Sarah Boyce]

Hi Akshat, I don't have a strong preference for the implementation direction at this point. If you have multiple ideas for addressing the regression, I'd encourage exploring them and comparing both their performance and complexity. It would be useful to see the trade-offs between the approaches before settling on one. Thank you!