id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
24670	Django migrations are not scaling well with number of operations in migration	Nikolay Kurevin	nobody	"In our project there are around 200 models, most of them are  in a single app.

I've been testing potencial performance improvements of migration runner in django 1.8.

When running migration that adds 17 fields this happens:

  Rendering model states... DONE (23.982s)
  Applying xxx... OK (28.930s)

50 seconds to add just 17 fields to a completely empty DB. 
Similar results shows current django master.

In comparison, migration that adds a single field:
  Rendering model states... DONE (2.994s)
  Applying xxx2... OK (1.540s)


Profiling of 17 fields migration:
  Rendering model states... DONE (27.341s)
  Applying xxx... OK (33.287s)

{{{
         26957269 function calls (26828616 primitive calls) in 61.414 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.007    0.007   61.414   61.414 django/core/management/commands/migrate.py:55(handle)
        1    0.000    0.000   60.634   60.634 django/db/migrations/executor.py:65(migrate)
      525    0.003    0.000   51.490    0.098 django/db/migrations/operations/fields.py:43(state_forwards)
      582    0.030    0.000   51.487    0.088 django/db/migrations/state.py:88(reload_model)
       54    0.016    0.000   39.538    0.732 django/db/migrations/state.py:240(render_multiple)
     6174    0.251    0.000   39.521    0.006 django/db/migrations/state.py:507(render)
        1    0.005    0.005   33.287   33.287 django/db/migrations/executor.py:132(apply_migration)
        1    0.055    0.055   33.260   33.260 django/db/migrations/migration.py:86(apply)
    15525    1.869    0.000   25.929    0.002 django/apps/registry.py:334(clear_cache)
       40    0.001    0.000   25.852    0.646 django/db/migrations/migration.py:72(mutate_state)
7889/6140    0.648    0.000   23.327    0.004 django/db/models/base.py:72(__new__)
   137224    0.725    0.000   22.687    0.000 django/db/migrations/state.py:476(construct_fields)
  2049773   19.569    0.000   19.569    0.000 django/db/models/options.py:711(_expire_cache)
     7889    0.040    0.000   13.726    0.002 django/db/migrations/state.py:273(register_model)
     7616    0.028    0.000   12.333    0.002 django/db/migrations/state.py:281(unregister_model)
    25408    0.135    0.000   10.822    0.000 django/db/models/fields/related.py:1867(deconstruct)
    25408    0.250    0.000   10.687    0.000 django/db/models/fields/related.py:1551(deconstruct)
    28344    5.208    0.000   10.398    0.000 django/db/models/fields/related.py:309(swappable_setting)
    43915    0.147    0.000    9.827    0.000 django/utils/lru_cache.py:94(wrapper)
    21213    1.501    0.000    9.416    0.000 django/apps/registry.py:153(get_models)
118321/106078    0.231    0.000    9.062    0.000 django/db/models/base.py:321(add_to_class)
  3985834    3.512    0.000    7.534    0.000 django/apps/config.py:164(get_models)
       19    0.001    0.000    7.105    0.374 django/db/migrations/state.py:142(clone)
       19    0.007    0.000    6.920    0.364 django/db/migrations/state.py:145(<dictcomp>)
     3572    0.056    0.000    6.913    0.002 django/db/migrations/state.py:496(clone)
   127520    4.815    0.000    5.706    0.000 django/db/models/fields/__init__.py:358(deconstruct)
     1891    0.024    0.000    5.029    0.003 django/db/models/fields/related.py:2518(contribute_to_class)
     1749    0.092    0.000    4.856    0.003 django/db/models/fields/related.py:2048(create_many_to_many_intermediary
_model)
   741163    2.485    0.000    3.776    0.000 /usr/lib/python2.7/collections.py:104(values)

}}}
We also have a initial migration that adds ~400 FK fields and it takes up to 10 minutes to execute."	Cleanup/optimization	closed	Migrations	dev	Normal	duplicate	migrations optimization	marten.knbk@… Nikolay Kurevin	Unreviewed	0	0	0	0	0	0