#26867 closed Cleanup/optimization (needsinfo)
Migrate very slow
| Reported by: | Hugo Tácito | Owned by: | nobody |
|---|---|---|---|
| Component: | Migrations | Version: | 1.9 |
| Severity: | Normal | Keywords: | |
| Cc: | Triage Stage: | Unreviewed | |
| Has patch: | no | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
When I have to run migrations between the change of modules it gets very slow.
Example:
Applying financial.010_auto_20150819_1451... OK (24.984s)
(wait 5 minutes)
Applying fleet.00120_auto_20150819_1451... OK (17.156s)
Applying fleet.00121_auto_20150819_1451... OK (10.234s)
Applying fleet.00122_auto_20150819_1451... OK (09.352s)
Applying fleet.00123_auto_20150819_1451... OK (14.471s)
(wait 5 minutes)
Applying parking.0054_auto_20150819_1451... OK (11.604s)
(wait 5 minutes)
Applying poll.0085_auto_20150805_1413... OK (9.664s)
Applying poll.0086_auto_20150819_1451... OK (34.831s)
(wait 5 minutes)
Applying protocol.0021_auto_20150819_1451... OK (10.068s)
I think this happens because after the first migration is executed, the 'migrate' command reloads all models again for each subsequent migration, without ignoring the migrations that was already run in previous 'migrate' commands.
Change History (5)
comment:1 by , 8 years ago
comment:2 by , 8 years ago
| Resolution: | → needsinfo |
|---|---|
| Status: | new → closed |
comment:3 by , 8 years ago
Hello tim,
I'm sorry for the lack of information but here is the zipped profile, https://dl.dropboxusercontent.com/u/31511/migrate2.pstats.gz
And the full out:
Operations to perform: Apply all migrations: projetos, clipping, ae, ponto, demandas, portaria, frota, enquete, temp_rh2, edu, financeiro, arquivo, estagios, avaliacao_integrada, authtoken, plan_v2, materiais, estacionamento, convenios, contratos, comum, contenttypes, remanejamento, almoxarifado, temp_ideias, rh, contracheques, protocolo, progressoes, sessions, reversion, chaves, pesquisa, auth, eleicao, cnpq, rsc, ferias, ldap_backend, processo_seletivo, admin, cpa, gestao, compras, orcamento, cursos, pdi, saude, centralservicos, planejamento, microsoft, patrimonio Running pre-migrate handlers for application auth Running pre-migrate handlers for application contenttypes Running pre-migrate handlers for application sessions Running pre-migrate handlers for application djtools Running pre-migrate handlers for application admin Running pre-migrate handlers for application django_extensions Running pre-migrate handlers for application pagination Running pre-migrate handlers for application django_tables2 Running pre-migrate handlers for application django_tables2_reports Running pre-migrate handlers for application rest_framework Running pre-migrate handlers for application rest_framework_swagger Running pre-migrate handlers for application authtoken Running pre-migrate handlers for application reversion Running pre-migrate handlers for application reversion_compare Running pre-migrate handlers for application comum Running pre-migrate handlers for application rh Running pre-migrate handlers for application almoxarifado Running pre-migrate handlers for application edu Running pre-migrate handlers for application gestao Running pre-migrate handlers for application patrimonio Running pre-migrate handlers for application ponto Running pre-migrate handlers for application centralservicos Running pre-migrate handlers for application contracheques Running pre-migrate handlers for application protocolo Running pre-migrate handlers for application frota Running pre-migrate handlers for application financeiro Running pre-migrate handlers for application ldap_backend Running pre-migrate handlers for application chaves Running pre-migrate handlers for application estacionamento Running pre-migrate handlers for application cursos Running pre-migrate handlers for application remanejamento Running pre-migrate handlers for application contratos Running pre-migrate handlers for application planejamento Running pre-migrate handlers for application convenios Running pre-migrate handlers for application orcamento Running pre-migrate handlers for application materiais Running pre-migrate handlers for application projetos Running pre-migrate handlers for application compras Running pre-migrate handlers for application ae Running pre-migrate handlers for application cnpq Running pre-migrate handlers for application pdi Running pre-migrate handlers for application cpa Running pre-migrate handlers for application processo_seletivo Running pre-migrate handlers for application saude Running pre-migrate handlers for application arquivo Running pre-migrate handlers for application microsoft Running pre-migrate handlers for application ferias Running pre-migrate handlers for application temp_ideias Running pre-migrate handlers for application eleicao Running pre-migrate handlers for application enquete Running pre-migrate handlers for application rsc Running pre-migrate handlers for application progressoes Running pre-migrate handlers for application estagios Running pre-migrate handlers for application portaria Running pre-migrate handlers for application pesquisa Running pre-migrate handlers for application avaliacao_integrada Running pre-migrate handlers for application clipping Running pre-migrate handlers for application temp_rh2 Running pre-migrate handlers for application plan_v2 Running pre-migrate handlers for application demandas Running migrations: Rendering model states... DONE (3.947s) Applying reversion.0002_auto_20141216_1509... FAKED (0.022s) Applying saude.0023_auto_20160630_1419... FAKED (0.043s) Running post-migrate handlers for application auth Running post-migrate handlers for application contenttypes Running post-migrate handlers for application sessions Running post-migrate handlers for application djtools Running post-migrate handlers for application admin Running post-migrate handlers for application django_extensions Running post-migrate handlers for application pagination Running post-migrate handlers for application django_tables2 Running post-migrate handlers for application django_tables2_reports Running post-migrate handlers for application rest_framework Running post-migrate handlers for application rest_framework_swagger Running post-migrate handlers for application authtoken Running post-migrate handlers for application reversion Running post-migrate handlers for application reversion_compare Running post-migrate handlers for application comum Running post-migrate handlers for application rh Running post-migrate handlers for application almoxarifado Running post-migrate handlers for application edu Running post-migrate handlers for application gestao Running post-migrate handlers for application patrimonio Running post-migrate handlers for application ponto Running post-migrate handlers for application centralservicos Running post-migrate handlers for application contracheques Running post-migrate handlers for application protocolo Running post-migrate handlers for application frota Running post-migrate handlers for application financeiro Running post-migrate handlers for application ldap_backend Running post-migrate handlers for application chaves Running post-migrate handlers for application estacionamento Running post-migrate handlers for application cursos Running post-migrate handlers for application remanejamento Running post-migrate handlers for application contratos Running post-migrate handlers for application planejamento Running post-migrate handlers for application convenios Running post-migrate handlers for application orcamento Running post-migrate handlers for application materiais Running post-migrate handlers for application projetos Running post-migrate handlers for application compras Running post-migrate handlers for application ae Running post-migrate handlers for application cnpq Running post-migrate handlers for application pdi Running post-migrate handlers for application cpa Running post-migrate handlers for application processo_seletivo Running post-migrate handlers for application saude Running post-migrate handlers for application arquivo Running post-migrate handlers for application microsoft Running post-migrate handlers for application ferias Running post-migrate handlers for application temp_ideias Running post-migrate handlers for application eleicao Running post-migrate handlers for application enquete Running post-migrate handlers for application rsc Running post-migrate handlers for application progressoes Running post-migrate handlers for application estagios Running post-migrate handlers for application portaria Running post-migrate handlers for application pesquisa Running post-migrate handlers for application avaliacao_integrada Running post-migrate handlers for application clipping Running post-migrate handlers for application temp_rh2 Running post-migrate handlers for application plan_v2 Running post-migrate handlers for application demandas python -m cProfile -o migrate2.pstats manage.py migrate --fake -v 3 696.72s user 11.25s system 98% cpu 12:01.12 total
comment:4 by , 8 years ago
Hi Hugo, this is not really surprising since that you have 100 migrations in some of the apps. Please squash them such that Django can optimize migration operations and use those instead of dozens of small changes.
comment:5 by , 8 years ago
Hello MarkusH, the problem is that my code is used by different companies with different databases, so I think it's hard to manage squashed migrations between all the companies. Although I solved my problem with this diff:
diff --git a/django/db/migrations/executor.py b/django/db/migrations/executor.py
index fb9c7a9..25170c3 100644
--- a/django/db/migrations/executor.py
+++ b/django/db/migrations/executor.py
@@ -135,6 +135,8 @@ class MigrationExecutor(object):
# Only mutate the state if the migration is actually applied
# to make sure the resulting state doesn't include changes
# from unrelated migrations.
+ if 'apps' in state.__dict__:
+ delattr(state, 'apps') # Remove the render all
migration.mutate_state(state, preserve=False)
applied_migrations.remove(migration)
Note:
See TracTickets
for help on using tickets.
Can you provide a sample project to reproduce, some profiling data that points to where the slow down is, or a patch for Django that improves things? If not, it seems difficult to turn what you've provided so far into a patch.