Opened 7 months ago

Last modified 3 weeks ago

#31169 assigned New feature

Allow parallel test runner to work with Windows/macOS `spawn` process start method.

Reported by: Brandon Navra Owned by: Ahmad A. Hussein
Component: Testing framework Version: master
Severity: Normal Keywords:
Cc: Abhijeet Viswa, Ichlasul Affan Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Brandon Navra)

Python 3.8 on MacOS has changed the default start method for the multiprocessing module from fork to spawn: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods.

When running tests with the --parallel flag, this causes the worker processes to fail with django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. as they no longer have a copy of the parent memory state. It can also cause the workers to fail to find the cloned dbs ( {{django.db.utils.OperationalError: FATAL: database "xxx_1" does not exist}} ) as the db test prefix is missing.

I have attached a patch which changes django.test.runner._init_worker (the worker initialiser for ParallelTestSuite) to run django.setup() and set the db name to one with the test_ prefix.

Attachments (1)

Ensure_Django_is_setup_correctly_in_parallel_test_workers.patch (1.2 KB) - added by Brandon Navra 7 months ago.

Download all attachments as: .zip

Change History (17)

comment:1 Changed 7 months ago by Brandon Navra

Description: modified (diff)

comment:2 Changed 7 months ago by felixxm

spawn() is also a default method on Windows, and we don't encounter any issues with it 🤔.

comment:3 Changed 7 months ago by Brandon Navra

I'm still trying to research the exact root cause. The Python issue which triggered this change has snippets of info: https://code.djangoproject.com/ticket/31169
but nothing conclusive. My theory is that the memory copying semantics between MacOS and Windows are different and hence the spawn method doesn't have identical behaviour between the two.

comment:4 Changed 7 months ago by felixxm

Ahhh, sorry we don't use parallel on Windows.

comment:5 Changed 7 months ago by Carlton Gibson

Summary: Parallel test mode not working on MacOS with recent versions of Python 3Allow parallel test runner to work with Windows/macOS `spawn` process start method.
Triage Stage: UnreviewedAccepted
Type: BugNew feature
Version: 3.0master

Parallel running is disabled on Windows:

https://github.com/django/django/blob/59b4e99dd00b9c36d56055b889f96885995e4240/django/test/runner.py#L286-L295

def default_test_processes():
    """Default number of test processes when using the --parallel option."""
    # The current implementation of the parallel test runner requires
    # multiprocessing to start subprocesses with fork().
    if multiprocessing.get_start_method() != 'fork':
        return 1
    try:
        return int(os.environ['DJANGO_TEST_PROCESSES'])
    except KeyError:
        return multiprocessing.cpu_count()

I'll accept this as a new feature: the limitation has been there since it was implemented.

Brandon, your patch is tiny. Is it really that simple? We'd need tests and a few other adjustments (like to the function above) but, fancy opening a PR?

Last edited 7 months ago by Carlton Gibson (previous) (diff)

comment:6 Changed 7 months ago by Carlton Gibson

So this occurs on macOS 10.15. (I have 10.14 currently so can't experiment there.)

Applying the patch on Windows, alas, doesn't immediately solve the issue, but it is INSTALLED_APPS/AppRegistry errors that are raised, so it's going to be in the right ball-park.

More investigating needed, but this would be a good one to land.

comment:7 Changed 7 months ago by Brandon Navra

I created a PR with the changes from my patch: https://github.com/django/django/pull/12321

FYI" I am on macOS 10.14.6

I'm not sure how best to adjust default_test_processes as i've always used the --parallel flag with a parameter.
Also, could you provide some guidance on how you'd like this tested

Last edited 7 months ago by Brandon Navra (previous) (diff)

comment:8 Changed 7 months ago by Carlton Gibson

FYI" I am on macOS 10.14.6

Super. I had a 3.7. env active. I can reproduce with Python 3.8.

comment:9 Changed 7 months ago by Brandon Navra

Has patch: set
Last edited 7 months ago by felixxm (previous) (diff)

comment:10 Changed 7 months ago by Thierry Bastian

Thanks for the report. I had the same issue but did not find the root cause in https://code.djangoproject.com/ticket/31116.
I would love to see that being resolved.

comment:11 Changed 6 months ago by Carlton Gibson

Patch needs improvement: set

comment:12 Changed 6 months ago by Peter Inglesby

I ran into this while running the Django test suite, and when applying the patch in PR 12321, I get the same problem with a different exception:

Traceback (most recent call last):
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
  File "/Users/inglesp/src/django/django/tests/fixtures_regress/tests.py", line 18, in <module>
    from .models import (
  File "/Users/inglesp/src/django/django/tests/fixtures_regress/models.py", line 1, in <module>
    from django.contrib.auth.models import User
  File "/Users/inglesp/src/django/django/django/contrib/auth/models.py", line 3, in <module>
    from django.contrib.contenttypes.models import ContentType
  File "/Users/inglesp/src/django/django/django/contrib/contenttypes/models.py", line 133, in <module>
    class ContentType(models.Model):
  File "/Users/inglesp/src/django/django/django/db/models/base.py", line 113, in __new__
    raise RuntimeError(
RuntimeError: Model class django.contrib.contenttypes.models.ContentType doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

I'm on OSX 10.15 with Python 3.8.

comment:13 Changed 5 months ago by Abhijeet Viswa

Cc: Abhijeet Viswa added

comment:14 Changed 4 months ago by Ichlasul Affan

Cc: Ichlasul Affan added

comment:15 Changed 7 weeks ago by Ahmad A. Hussein

Has patch: unset
Owner: changed from nobody to Ahmad A. Hussein
Patch needs improvement: unset
Status: newassigned

comment:16 Changed 3 weeks ago by Adam (Chainz) Johnson

For those looking for a workaround, here's how to add the appropriate call to reset back to fork mode: https://adamj.eu/tech/2020/07/21/how-to-use-djangos-parallel-testing-on-macos-with-python-3.8-plus/

Note: See TracTickets for help on using tickets.
Back to Top