Django 1.8+ Migrations Hold Onto Way More Memory

[James Pulec]

After bumping to 1.8, rendering model states when running migrate seems to consume a lot more memory. Somewhere on the order of 4x-6x more. It doesn't look like this memory is properly released back to the OS after running migrate, and just hangs around.

I discovered this calling manage.py test when using a custom test runner that makes a subprocess call, and was unable to do so, because the python process was already consuming so much memory that os.fork() couldn't create a new process.

[Tim Graham]

Did you test with 1.8.1 after #24591? The solution for that ticket is a bit different on master, so testing there for comparison would also be useful.

[James Pulec]

Testing with 1.8.1 seems to yield the same results. Somewhere around 1GB of virtual memory is being held after rendering the model migrations, and then my test runner fails to fork. However, running with the flag --keepdb runs smoothly as I would expect.

I'm working on trying to run against master, but I have so many dependencies that break against master that it might take some time.

[James Pulec]

For reference, I finally managed to run my tests against master. They pass, but I think the margin is pretty slim. The VM I do all my testing in has 2GB of RAM. On 1.8 and 1.8.1 my running test holds onto ~47% of my ram according to top. On master that number comes down closer to 37% so it runs for my use case.

However, it still shows that running all those migrations on creation of a test database gobbles up all that ram and doesn't seem to release it right away.

[Matt Barr]

I've encountered the exact same problem. Our virtual machines used for development would require 2GB of memory just to run migrations (as opposed to the 512 they use now). We cannot upgrade until this is resolved.

[Markus Holtermann]

Could you give the pull request ​https://github.com/django/django/pull/5178 a try.

[Markus Holtermann <info@…>]

In 5aa55038:

Fixed #24743, #24745 -- Optimized migration plan handling

The change partly goes back to the old behavior for forwards migrations
which should reduce the amount of memory consumption (#24745). However,
by the way the current state computation is done (there is no
state_backwards on a migration class) this change cannot be applied to
backwards migrations. Hence rolling back migrations still requires the
precomputation and storage of the intermediate migration states.

This improvement also implies that Django does not handle mixed
migration plans anymore. Mixed plans consist of a list of migrations
where some are being applied and others are being unapplied.

Thanks Andrew Godwin, Josh Smeaton and Tim Graham for the review as well
as everybody involved on the ticket that kept me looking into the issue.