Opened 4 years ago

Last modified 4 years ago

#24396 new Cleanup/optimization

Migrate only required models to the "other" database

Reported by: Aymeric Augustin Owned by: nobody
Component: Testing framework Version: master
Severity: Normal Keywords:
Cc: Josh Smeaton, Michael Manfre Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

It seems wasteful to create hundreds of tables on the "other" database when only a few tests of Django's test suite touch it.

I'm attaching a patch optimizing this. However it doesn't appear to save much time in practice. "Rendering model states" is the killer.

Attachments (1)

24396.patch (2.3 KB) - added by Aymeric Augustin 4 years ago.

Download all attachments as: .zip

Change History (5)

Changed 4 years ago by Aymeric Augustin

Attachment: 24396.patch added

comment:1 Changed 4 years ago by Aymeric Augustin

Cc: Josh Smeaton added

A quick test shows that it takes 49 seconds to start running the tests with this patch vs. 53 seconds without on Postgres.

On SQLite the difference isn't measurable.

comment:2 Changed 4 years ago by Josh Smeaton

Thanks for knocking this up. I just ran the the test suite on Oracle with this patch:

1minute 53 seconds to create the first database
27 seconds to create the second database

(The cleanup/dropping databases is also heavily weighted against the first database - I didn't measure but it would have been around 1 minute and 30 seconds respectively)

Without Patch:

2 minutes to create the first database
1 minute 35 seconds to create the second database.

About 1 minute is saved on the creation of the second database. I suspect mssql will see similar improvements, so I'll tag manfre here incase he wants to check it out.

I have done some measurements WRT Oracle and migrations but I abandoned them. Querying for table_names, view_names, and constraints takes quite a long time, and the results are never cached (clearing the cache wasn't as straightforward as it could have been).

https://github.com/jarshwah/django/tree/migrate-other-tests incase others struggle with git apply x.patch like I did :P

comment:3 Changed 4 years ago by Josh Smeaton

Cc: Michael Manfre added

comment:4 Changed 4 years ago by Michael Manfre

Triage Stage: UnreviewedAccepted

Wall timing of test runs:

Using jarshwah's branch with this patch,

Command line to start of create "other" database is 2 minutes.
Command line to start of first test is 3 minutes 30 seconds.

Using latest master,

Command line to start of create "other" database is 2 minutes 1 second.
Command line to start of first test is 3 minutes 50 seconds.

~20 seconds speed up. This is a worthwhile improvement that will add up to a lot of saved time based upon how frequently I run the test suite.

Note: See TracTickets for help on using tickets.
Back to Top