Changes between Initial Version and Version 2 of Ticket #31804


Ignore:
Timestamp:
Jul 20, 2020, 10:31:40 AM (4 years ago)
Author:
Ahmad A. Hussein
Comment:

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #31804

    • Property Owner changed from nobody to Ahmad A. Hussein

  • Ticket #31804 – Description

    initial v2  
    11Parallelizing database cloning processes would yield a nice speed-up for running Django's own test suite (and all django projects that use the default test runner)
    22
    3 So far there are two main ways I see we can implement this:
    4  - Use existing backend utilities e.g mysqlpump instead of mysqldump
    5  - Use a normal multiprocessing pool on top of our existing cloning code
     3So far there are three main ways I see we can implement this:
     4 - Use a multiprocessing pool at the setup_databases level that'll create workers which run ```clone_test_db``` for each method
     5 - Use a pool at the ```clone_test_db``` level which parallelizes the internal ```_clone_test_db``` call
     6 - Scrap parallelizing the cloning in general, but parallelizing the internals of specific backends (at least MySQL fits here)
    67
     8In the first two options, we'd have to refactor MySQL's cloning process since it has another call to ```_clone_db```. We have to because otherwise we'd have a dump being created inside of each parallel process, slowing the workers greatly.
     9
     10In the last option, we could consider using mysqlpump instead of mysqldump for both exporting the database and restoring it. The con of this approach is that it isn't general enough to apply to the other backends.
     11
     12Oracle's cloning process(although not merged in the current master) has internal support for option 3 (users can specify a PARALLEL variable to speed-up expdp/impdp utilities), and it can also use the first two options.
     13
     14The major con though with the first two options is forcing parallelization
     15
Back to Top