Parallel tests fail on Python 3.11+ when IsolatedAsyncioTestCase is used

[Matt Hegarty]

Parallel tests work in python3.10 but not in 3.11 or 3.12. Running tests in non-parallel mode works fine.

./manage.py test --parallel   

Error is:

TypeError: cannot pickle '_contextvars.Context' object

See attached stack trace.

• django 4.2.6

• python3.10.13 (parallel ok)
• python3.11.6 (parallel fails)
• python3.12.0 (parallel fails)

MacBook Pro Ventura 13.4.1

Similar to https://code.djangoproject.com/ticket/34010

The database is Postgres:13 running in a Docker container

[Matt Hegarty]

stack trace

[Mariusz Felisiak]

Thanks for this report. However, as far as I'm aware, Python versions shipped with homebrew have various issues and it's not recommended to use. I don't think you've explained the issue in enough detail to confirm a bug in Django. Folks are running our test suite on MacOS without any issues. This also has nothing to do with #34010. Moreover, Django 4.2 doesn't officially support Python 3.12 yet.

Please reopen the ticket if you can debug your issue and provide details about why and where Django is at fault. You can try to use one of support channels where will help you debug your issue.

[Matt Hegarty]

• Reproducible on Ubuntu and MacOS
• using python 3.11 or 3.12 (3.10 is ok)
• using Django 4.1 or higher (4.0.10 is ok)
• Needs to have 2 or more test classes
• 1 class has to subclass IsolatedAsyncioTestCase

This will show the issue:

git clone git@github.com:matthewhegarty/tutorial.git
cd tutorial
mkvirtualenv -p `which python3.11` -r requirements.txt tutorial-311

# fails
./manage.py test --parallel

# ok
./manage.py test

Some other testing

• Django 4.0.10 [OK]
• Django 4.1 [FAIL] (but with different error - see attached)
• Django 4.1.1 [FAIL] (error from https://code.djangoproject.com/ticket/34010)
• Django 4.1.2 [FAIL] (first instance of this error)
• Django 4.1.8 [FAIL]

[Matt Hegarty]

stack trace from 4.0.1

[Natalia Bidart]

I can confirm I also get the reported error with the given reproducer, running on a Manjaro system (amd64):

Traceback (most recent call last):
  File "/home/nessita/fellowship/django/tests/runtests.py", line 783, in <module>
    failures = django_tests(
               ^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/tests/runtests.py", line 421, in django_tests
    failures = test_runner.run_tests(test_labels)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/django/test/runner.py", line 1068, in run_tests
    result = self.run_suite(suite)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/django/test/runner.py", line 995, in run_suite
    return runner.run(suite)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/runner.py", line 217, in run
    test(result)
  File "/usr/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/django/test/runner.py", line 541, in run
    subsuite_index, events = test_results.next(timeout=0.1)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_contextvars.Context' object
Exception ignored in: <function Pool.__del__ at 0x7f16d6958680>
Traceback (most recent call last):
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 268, in __del__
ResourceWarning: unclosed running multiprocessing pool <multiprocessing.pool.Pool state=RUN pool_size=4>

What I don't have clarity is whether this ever worked, I have bisected up to the adding of ASGI support (commit a415ce70bef6d91036b00dd2c8544aed7aeeaaed) and the given tests are still failing. I have also checked out stable/4.0.x and the tests are still not working.

Matt, can you please explain how Django is at fault here? It seems that IsolatedAsyncioTestCase hasn't been (ever?) supported?

[Matt Hegarty]

I have also checked out stable/4.0.x and the tests are still not working.

• The tests pass for me when running against stable/4.0.x (python 3.11) (i.e. no crash)
• I have updated the ​test project to see if you can reproduce

[Natalia Bidart]

Replying to Matt Hegarty:

• The tests pass for me when running against stable/4.0.x (python 3.11) (i.e. no crash)
• I have updated the ​test project to see if you can reproduce

Thanks Matt for the extra details but it fails for me with 4.0.x:

[~/fellowship/tutorial main|✚ 1…1]$ python -c "import django; print(django.get_version())"
4.0.11

[~/fellowship/tutorial main|✚ 1…1]$ ./manage.py test --parallel
Found 4 test(s).
System check identified no issues (0 silenced).
Traceback (most recent call last):
  File "/home/nessita/fellowship/tutorial/./manage.py", line 22, in <module>
    main()
  File "/home/nessita/fellowship/tutorial/./manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/commands/test.py", line 24, in run_from_argv
    super().run_from_argv(argv)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/base.py", line 414, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/commands/test.py", line 68, in handle
    failures = test_runner.run_tests(test_labels)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/test/runner.py", line 1006, in run_tests
    result = self.run_suite(suite)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/test/runner.py", line 930, in run_suite
    return runner.run(suite)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/runner.py", line 217, in run
    test(result)
  File "/usr/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/test/runner.py", line 493, in run
    subsuite_index, events = test_results.next(timeout=0.1)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_contextvars.Context' object

[Matt Hegarty]

I wonder why it reports that there are 4 tests in your test run.

My output (Mac Ventura) using homebrew is below (tests pass). However, I tried the same on my Ubuntu instance, and it failed with the same error you got. Mariusz stated that homebrew is dodgy, so perhaps that's the reason for the success.

I guess no need to spend any more time on it and we can leave it closed.

(tutorial-311) ➜  tutorial git:(main) ✗ python -c "import django; print(django.get_version())"
4.0.11
(tutorial-311) ➜  tutorial git:(main) ✗ ./manage.py test --parallel
Found 2 test(s).
System check identified no issues (0 silenced).
..
----------------------------------------------------------------------
Ran 2 tests in 0.005s

OK

[Matt Hegarty]

one final thought, the 'tutorial' test app passes when using python3.10 on Ubuntu (any version of django) - it might be worth you confirming that is the same for you.

[Natalia Bidart]

I can confirm that the ./manage.py test --parallel works with any version of Django (4.0, 4.2, 5.0b1) and Python 3.10.

root@568680bc51ec:/tutorial# python --version
Python 3.10.13
root@568680bc51ec:/tutorial# python manage.py test --parallel
Found 2 test(s).
System check identified no issues (0 silenced).
..
----------------------------------------------------------------------
Ran 2 tests in 0.025s

OK

Given that this is an issue related to the Python version, I'm not sure how to proceed. Mariusz, Carlton, what would be your advice in this case?

[Carlton Gibson]

So the tests don't run in parallel on macOS prior to Django 4.1 (​release notes) so the whole pickle step never gets hit. I'd guess that would explain the issue there.

It seems that IsolatedAsyncioTestCase hasn't been (ever?) supported?

That's the nub of it.

Django's SimpleTestCase added equivalent support for async def test methods in Django 3.1 (​docs), but IsolatedAsyncioTestCase wasn't part of that.

This ticket is maybe a feature request to add support for IsolatedAsyncioTestCase subclasses but what's the need, and given the error…

TypeError: cannot pickle '_contextvars.Context' object

… is it going to be feasible at all?

(It might be but, for me, I'd like some indication of a yes there before accepting such, so we don't send folks down a dead-end rabbit hole.)

[Matt Hegarty]

Just to clarify, the tests pass with any version of Django if you use python3.10 (tested on MacOS and Ubuntu). They fail when running 3.11 or 3.12.

[Carlton Gibson]

Replying to Matt Hegarty:

Just to clarify, the tests pass with any version of Django if you use python3.10 (tested on MacOS and Ubuntu). They fail when running 3.11 or 3.12.

I guess the question then is ​what change led to this in Python 3.11?

The addition of IsolatedAsyncioTestCase.enterAsyncContext() is a ​likely candidate.

But, still, use of IsolatedAsyncioTestCase has never been supported by Django, or tested in its test suite, so any success prior to 3.11 would be coincidental (leveraging undefined behaviour from Django's point of view). Adding such support would still be a new feature.

[Matt Hegarty]

OK. Thanks all for investigating.

[Natalia Bidart]

Setting to wontfix since that resolution aligns better with the latest outcome. Thanks everyone!