Allow parallel test runner to work with Windows/macOS `spawn` process start method.

[Brandon Navra]

Python 3.8 on MacOS has changed the default start method for the multiprocessing module from fork to spawn: ​https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods.

When running tests with the --parallel flag, this causes the worker processes to fail with django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. as they no longer have a copy of the parent memory state. It can also cause the workers to fail to find the cloned dbs ( {{django.db.utils.OperationalError: FATAL: database "xxx_1" does not exist}} ) as the db test prefix is missing.

I have attached a patch which changes django.test.runner._init_worker (the worker initialiser for ParallelTestSuite) to run django.setup() and set the db name to one with the test_ prefix.

[Mariusz Felisiak]

spawn() is also a default method on Windows, and we don't encounter any issues with it 🤔.

[Brandon Navra]

I'm still trying to research the exact root cause. The Python issue which triggered this change has snippets of info: https://code.djangoproject.com/ticket/31169
but nothing conclusive. My theory is that the memory copying semantics between MacOS and Windows are different and hence the spawn method doesn't have identical behaviour between the two.

[Mariusz Felisiak]

Ahhh, sorry we don't use parallel on Windows.

[Carlton Gibson]

Parallel running is disabled on Windows:

​https://github.com/django/django/blob/59b4e99dd00b9c36d56055b889f96885995e4240/django/test/runner.py#L286-L295

def default_test_processes():
    """Default number of test processes when using the --parallel option."""
    # The current implementation of the parallel test runner requires
    # multiprocessing to start subprocesses with fork().
    if multiprocessing.get_start_method() != 'fork':
        return 1
    try:
        return int(os.environ['DJANGO_TEST_PROCESSES'])
    except KeyError:
        return multiprocessing.cpu_count()

I'll accept this as a new feature: the limitation has been there since it was implemented.

Brandon, your patch is tiny. Is it really that simple? We'd need tests and a few other adjustments (like to the function above) but, fancy opening a PR?

[Carlton Gibson]

So this occurs on macOS 10.15. (I have 10.14 currently so can't experiment there.)

Applying the patch on Windows, alas, doesn't immediately solve the issue, but it is INSTALLED_APPS/AppRegistry errors that are raised, so it's going to be in the right ball-park.

More investigating needed, but this would be a good one to land.

[Brandon Navra]

I created a PR with the changes from my patch: ​https://github.com/django/django/pull/12321

FYI" I am on macOS 10.14.6

I'm not sure how best to adjust default_test_processes as i've always used the --parallel flag with a parameter.
Also, could you provide some guidance on how you'd like this tested

[Carlton Gibson]

FYI" I am on macOS 10.14.6

Super. I had a 3.7. env active. I can reproduce with Python 3.8.

[Brandon Navra]

​PR

[Thierry Bastian]

Thanks for the report. I had the same issue but did not find the root cause in https://code.djangoproject.com/ticket/31116.
I would love to see that being resolved.

[Peter Inglesby]

I ran into this while running the Django test suite, and when applying the patch in ​PR 12321, I get the same problem with a different exception:

Traceback (most recent call last):
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
  File "/Users/inglesp/src/django/django/tests/fixtures_regress/tests.py", line 18, in <module>
    from .models import (
  File "/Users/inglesp/src/django/django/tests/fixtures_regress/models.py", line 1, in <module>
    from django.contrib.auth.models import User
  File "/Users/inglesp/src/django/django/django/contrib/auth/models.py", line 3, in <module>
    from django.contrib.contenttypes.models import ContentType
  File "/Users/inglesp/src/django/django/django/contrib/contenttypes/models.py", line 133, in <module>
    class ContentType(models.Model):
  File "/Users/inglesp/src/django/django/django/db/models/base.py", line 113, in __new__
    raise RuntimeError(
RuntimeError: Model class django.contrib.contenttypes.models.ContentType doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

I'm on OSX 10.15 with Python 3.8.

[Abhijeet Viswa]

An attempt at a fix: ​https://github.com/django/django/pull/12547

[Adam Johnson]

For those looking for a workaround, here's how to add the appropriate call to reset back to fork mode: ​https://adamj.eu/tech/2020/07/21/how-to-use-djangos-parallel-testing-on-macos-with-python-3.8-plus/

[Carlton Gibson]

PR is working nicely on macOS but needs a rebase, and a refactoring for review comments.

[Carlton Gibson]

A few issues on the PR to resolve, but looking very promising: 39.813s vs 249.146s on macOS, which is a bit of a speed-up 🙂

[Mariusz Felisiak <felisiak.mariusz@…>]

In ae91ecf6:

Refs #31169 -- Added DatabaseCreation.setup_worker_connection() hook.

[GitHub <noreply@…>]

In 795da630:

Refs #31169 -- Prevented infinite loop in tests on failures.

Regression in ae91ecf6a1037fb67d14841b66ac19d4c2ccc4ac.

[Carlton Gibson <carlton@…>]

In 3b3f38b:

Fixed #31169 -- Adapted the parallel test runner to use spawn.

Co-authored-by: Valz <ahmadahussein0@…>
Co-authored-by: Nick Pope <nick@…>

[Mariusz Felisiak <felisiak.mariusz@…>]

In ba298a32:

Refs #31169 -- Prevented infinite loop in parallel tests with custom test runner when using spawn.

Regression in 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.

Co-Authored-By: Mariusz Felisiak <felisiak.mariusz@…>

[GitHub <noreply@…>]

In 20d575b:

Refs #31169 -- Skipped test_get_test_db_clone_settings_not_supported on not in-memory SQLite database.

multiprocessing's start method is checked only for in-memory SQLite
databases.