Opened 4 years ago

Closed 3 years ago

#24745 closed Cleanup/optimization (fixed)

Django 1.8+ Migrations Hold Onto Way More Memory

Reported by: James Pulec Owned by: Markus Holtermann
Component: Migrations Version: 1.8
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

After bumping to 1.8, rendering model states when running migrate seems to consume a lot more memory. Somewhere on the order of 4x-6x more. It doesn't look like this memory is properly released back to the OS after running migrate, and just hangs around.

I discovered this calling manage.py test when using a custom test runner that makes a subprocess call, and was unable to do so, because the python process was already consuming so much memory that os.fork() couldn't create a new process.

Change History (8)

comment:1 Changed 4 years ago by Tim Graham

Did you test with 1.8.1 after #24591? The solution for that ticket is a bit different on master, so testing there for comparison would also be useful.

comment:2 Changed 4 years ago by James Pulec

Testing with 1.8.1 seems to yield the same results. Somewhere around 1GB of virtual memory is being held after rendering the model migrations, and then my test runner fails to fork. However, running with the flag --keepdb runs smoothly as I would expect.

I'm working on trying to run against master, but I have so many dependencies that break against master that it might take some time.

comment:3 Changed 4 years ago by Tim Graham

Triage Stage: UnreviewedAccepted
Type: BugCleanup/optimization

comment:4 Changed 4 years ago by James Pulec

For reference, I finally managed to run my tests against master. They pass, but I think the margin is pretty slim. The VM I do all my testing in has 2GB of RAM. On 1.8 and 1.8.1 my running test holds onto ~47% of my ram according to top. On master that number comes down closer to 37% so it runs for my use case.

However, it still shows that running all those migrations on creation of a test database gobbles up all that ram and doesn't seem to release it right away.

comment:5 in reply to:  4 Changed 4 years ago by Matt Barr

I've encountered the exact same problem. Our virtual machines used for development would require 2GB of memory just to run migrations (as opposed to the 512 they use now). We cannot upgrade until this is resolved.

comment:6 Changed 4 years ago by Markus Holtermann

Has patch: set
Owner: changed from nobody to Markus Holtermann
Status: newassigned

Could you give the pull request https://github.com/django/django/pull/5178 a try.

comment:7 Changed 4 years ago by Tim Graham

Triage Stage: AcceptedReady for checkin

comment:8 Changed 3 years ago by Markus Holtermann <info@…>

Resolution: fixed
Status: assignedclosed

In 5aa55038:

Fixed #24743, #24745 -- Optimized migration plan handling

The change partly goes back to the old behavior for forwards migrations
which should reduce the amount of memory consumption (#24745). However,
by the way the current state computation is done (there is no
state_backwards on a migration class) this change cannot be applied to
backwards migrations. Hence rolling back migrations still requires the
precomputation and storage of the intermediate migration states.

This improvement also implies that Django does not handle mixed
migration plans anymore. Mixed plans consist of a list of migrations
where some are being applied and others are being unapplied.

Thanks Andrew Godwin, Josh Smeaton and Tim Graham for the review as well
as everybody involved on the ticket that kept me looking into the issue.

Note: See TracTickets for help on using tickets.
Back to Top