Django migrations are not scaling well with number of operations in migration

[Nikolay Kurevin]

In our project there are around 200 models, most of them are in a single app.

I've been testing potencial performance improvements of migration runner in django 1.8.

When running migration that adds 17 fields this happens:

Rendering model states... DONE (23.982s)
Applying xxx... OK (28.930s)

50 seconds to add just 17 fields to a completely empty DB.
Similar results shows current django master.

In comparison, migration that adds a single field:

Rendering model states... DONE (2.994s)
Applying xxx2... OK (1.540s)

Profiling of 17 fields migration:

Rendering model states... DONE (27.341s)
Applying xxx... OK (33.287s)

         26957269 function calls (26828616 primitive calls) in 61.414 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.007    0.007   61.414   61.414 django/core/management/commands/migrate.py:55(handle)
        1    0.000    0.000   60.634   60.634 django/db/migrations/executor.py:65(migrate)
      525    0.003    0.000   51.490    0.098 django/db/migrations/operations/fields.py:43(state_forwards)
      582    0.030    0.000   51.487    0.088 django/db/migrations/state.py:88(reload_model)
       54    0.016    0.000   39.538    0.732 django/db/migrations/state.py:240(render_multiple)
     6174    0.251    0.000   39.521    0.006 django/db/migrations/state.py:507(render)
        1    0.005    0.005   33.287   33.287 django/db/migrations/executor.py:132(apply_migration)
        1    0.055    0.055   33.260   33.260 django/db/migrations/migration.py:86(apply)
    15525    1.869    0.000   25.929    0.002 django/apps/registry.py:334(clear_cache)
       40    0.001    0.000   25.852    0.646 django/db/migrations/migration.py:72(mutate_state)
7889/6140    0.648    0.000   23.327    0.004 django/db/models/base.py:72(__new__)
   137224    0.725    0.000   22.687    0.000 django/db/migrations/state.py:476(construct_fields)
  2049773   19.569    0.000   19.569    0.000 django/db/models/options.py:711(_expire_cache)
     7889    0.040    0.000   13.726    0.002 django/db/migrations/state.py:273(register_model)
     7616    0.028    0.000   12.333    0.002 django/db/migrations/state.py:281(unregister_model)
    25408    0.135    0.000   10.822    0.000 django/db/models/fields/related.py:1867(deconstruct)
    25408    0.250    0.000   10.687    0.000 django/db/models/fields/related.py:1551(deconstruct)
    28344    5.208    0.000   10.398    0.000 django/db/models/fields/related.py:309(swappable_setting)
    43915    0.147    0.000    9.827    0.000 django/utils/lru_cache.py:94(wrapper)
    21213    1.501    0.000    9.416    0.000 django/apps/registry.py:153(get_models)
118321/106078    0.231    0.000    9.062    0.000 django/db/models/base.py:321(add_to_class)
  3985834    3.512    0.000    7.534    0.000 django/apps/config.py:164(get_models)
       19    0.001    0.000    7.105    0.374 django/db/migrations/state.py:142(clone)
       19    0.007    0.000    6.920    0.364 django/db/migrations/state.py:145(<dictcomp>)
     3572    0.056    0.000    6.913    0.002 django/db/migrations/state.py:496(clone)
   127520    4.815    0.000    5.706    0.000 django/db/models/fields/__init__.py:358(deconstruct)
     1891    0.024    0.000    5.029    0.003 django/db/models/fields/related.py:2518(contribute_to_class)
     1749    0.092    0.000    4.856    0.003 django/db/models/fields/related.py:2048(create_many_to_many_intermediary
_model)
   741163    2.485    0.000    3.776    0.000 /usr/lib/python2.7/collections.py:104(values)

We also have a initial migration that adds ~400 FK fields and it takes up to 10 minutes to execute.

[Marten Kenbeek]

Could you elaborate on what potential performance improvements you have been testing? I've been working on some patches myself (see #24397, #24590, #24591), but I'd be interested to know if there are any potential optimizations that I've missed.

Thanks a lot for taking the time and effort to look into this, it is much appreciated.

[Nikolay Kurevin]

knbk, thanks for the useful links!
I was actually looking for performance improvements over Django 1.7.7, which we are currently using. Our 1.7.7 has few major optimizations backported from early 1.8, but it still slow, we might try to backport some of mentioned tickets.

Just a guess if you are looking for more improvements - it seems that runner does not cache inter-operation states, given that rendering time and migration running time match quite closely, and there is a lot of calls to state_forwards of AddField operation, 30 times more than actual number of operations, similarly the numbers of calls to render_multiple is about 3 times more, but it might be just previous migrations.
It would also be cool if django would not use inter-operation states at all or would allow some kind of merged operation, that would not waste time to compute many states for every small change.

[Markus Holtermann]

Hi kurevin, according to the trace you gave above this looks like #24591 which is about to be committed to master and in a slightly modified form to stable/1.8.x.

Regarding the backports to 1.7: there are two reasons those changes are not being backported:

1. in 1.8 we now cache the project state across migrations as much as possible and due to API changes many patches are not easy to be backported
2. 1.7 only receives security and data loss related fixes.

[Tim Graham]

Closing as a duplicate per Markus's comment. Please open separate tickets if there are other ideas for optimizations.