Opened 13 months ago

Closed 13 months ago

Last modified 12 months ago

#34930 closed Bug (wontfix)

Parallel tests fail on Python 3.11+ when IsolatedAsyncioTestCase is used

Reported by: Matt Hegarty Owned by: nobody
Component: Testing framework Version: 4.2
Severity: Normal Keywords: pickle _contextvars.Context _contextvars Context
Cc: Carlton Gibson, Mariusz Felisiak Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Parallel tests work in python3.10 but not in 3.11 or 3.12. Running tests in non-parallel mode works fine.

./manage.py test --parallel   


Error is:

TypeError: cannot pickle '_contextvars.Context' object

See attached stack trace.

  • django 4.2.6
  • python3.10.13 (parallel ok)
  • python3.11.6 (parallel fails)
  • python3.12.0 (parallel fails)

MacBook Pro Ventura 13.4.1

Similar to https://code.djangoproject.com/ticket/34010

The database is Postgres:13 running in a Docker container

Attachments (2)

st.txt (3.6 KB ) - added by Matt Hegarty 13 months ago.
stack trace
django_4_1_0_issue.txt (21.8 KB ) - added by Matt Hegarty 13 months ago.
stack trace from 4.0.1

Download all attachments as: .zip

Change History (18)

by Matt Hegarty, 13 months ago

Attachment: st.txt added

stack trace

comment:1 by Mariusz Felisiak, 13 months ago

Resolution: needsinfo
Status: newclosed
Summary: Parallel tests fail in python 3.11 and 3.12Parallel tests fail on Python 3.11+ and MacOS.

Thanks for this report. However, as far as I'm aware, Python versions shipped with homebrew have various issues and it's not recommended to use. I don't think you've explained the issue in enough detail to confirm a bug in Django. Folks are running our test suite on MacOS without any issues. This also has nothing to do with #34010. Moreover, Django 4.2 doesn't officially support Python 3.12 yet.

Please reopen the ticket if you can debug your issue and provide details about why and where Django is at fault. You can try to use one of support channels where will help you debug your issue.

comment:2 by Matt Hegarty, 13 months ago

  • Reproducible on Ubuntu and MacOS
  • using python 3.11 or 3.12 (3.10 is ok)
  • using Django 4.1 or higher (4.0.10 is ok)
  • Needs to have 2 or more test classes
  • 1 class has to subclass IsolatedAsyncioTestCase

This will show the issue:

git clone git@github.com:matthewhegarty/tutorial.git
cd tutorial
mkvirtualenv -p `which python3.11` -r requirements.txt tutorial-311

# fails
./manage.py test --parallel

# ok
./manage.py test


Some other testing

  • Django 4.0.10 [OK]
  • Django 4.1 [FAIL] (but with different error - see attached)
  • Django 4.1.1 [FAIL] (error from https://code.djangoproject.com/ticket/34010)
  • Django 4.1.2 [FAIL] (first instance of this error)
  • Django 4.1.8 [FAIL]
Last edited 13 months ago by Matt Hegarty (previous) (diff)

comment:3 by Matt Hegarty, 13 months ago

Resolution: needsinfo
Status: closednew

by Matt Hegarty, 13 months ago

Attachment: django_4_1_0_issue.txt added

stack trace from 4.0.1

comment:4 by Natalia Bidart, 13 months ago

Resolution: needsinfo
Status: newclosed

I can confirm I also get the reported error with the given reproducer, running on a Manjaro system (amd64):

Traceback (most recent call last):
  File "/home/nessita/fellowship/django/tests/runtests.py", line 783, in <module>
    failures = django_tests(
               ^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/tests/runtests.py", line 421, in django_tests
    failures = test_runner.run_tests(test_labels)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/django/test/runner.py", line 1068, in run_tests
    result = self.run_suite(suite)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/django/test/runner.py", line 995, in run_suite
    return runner.run(suite)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/runner.py", line 217, in run
    test(result)
  File "/usr/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/fellowship/django/django/test/runner.py", line 541, in run
    subsuite_index, events = test_results.next(timeout=0.1)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_contextvars.Context' object
Exception ignored in: <function Pool.__del__ at 0x7f16d6958680>
Traceback (most recent call last):
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 268, in __del__
ResourceWarning: unclosed running multiprocessing pool <multiprocessing.pool.Pool state=RUN pool_size=4>

What I don't have clarity is whether this ever worked, I have bisected up to the adding of ASGI support (commit a415ce70bef6d91036b00dd2c8544aed7aeeaaed) and the given tests are still failing. I have also checked out stable/4.0.x and the tests are still not working.

Matt, can you please explain how Django is at fault here? It seems that IsolatedAsyncioTestCase hasn't been (ever?) supported?

comment:5 by Matt Hegarty, 13 months ago

I have also checked out stable/4.0.x and the tests are still not working.

  • The tests pass for me when running against stable/4.0.x (python 3.11) (i.e. no crash)
  • I have updated the test project to see if you can reproduce

comment:6 by Matt Hegarty, 13 months ago

Summary: Parallel tests fail on Python 3.11+ and MacOS.Parallel tests fail on Python 3.11+ and django 4.1.0+ when IsolatedAsyncioTestCase is used

in reply to:  5 comment:7 by Natalia Bidart, 13 months ago

Replying to Matt Hegarty:

  • The tests pass for me when running against stable/4.0.x (python 3.11) (i.e. no crash)
  • I have updated the test project to see if you can reproduce

Thanks Matt for the extra details but it fails for me with 4.0.x:

[~/fellowship/tutorial main|✚ 1…1]$ python -c "import django; print(django.get_version())"
4.0.11

[~/fellowship/tutorial main|✚ 1…1]$ ./manage.py test --parallel
Found 4 test(s).
System check identified no issues (0 silenced).
Traceback (most recent call last):
  File "/home/nessita/fellowship/tutorial/./manage.py", line 22, in <module>
    main()
  File "/home/nessita/fellowship/tutorial/./manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/commands/test.py", line 24, in run_from_argv
    super().run_from_argv(argv)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/base.py", line 414, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/core/management/commands/test.py", line 68, in handle
    failures = test_runner.run_tests(test_labels)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/test/runner.py", line 1006, in run_tests
    result = self.run_suite(suite)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/test/runner.py", line 930, in run_suite
    return runner.run(suite)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/runner.py", line 217, in run
    test(result)
  File "/usr/lib/python3.11/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nessita/.virtualenvs/djangodev/lib/python3.11/site-packages/django/test/runner.py", line 493, in run
    subsuite_index, events = test_results.next(timeout=0.1)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_contextvars.Context' object

comment:8 by Matt Hegarty, 13 months ago

I wonder why it reports that there are 4 tests in your test run.

My output (Mac Ventura) using homebrew is below (tests pass). However, I tried the same on my Ubuntu instance, and it failed with the same error you got. Mariusz stated that homebrew is dodgy, so perhaps that's the reason for the success.

I guess no need to spend any more time on it and we can leave it closed.

(tutorial-311) ➜  tutorial git:(main) ✗ python -c "import django; print(django.get_version())"
4.0.11
(tutorial-311) ➜  tutorial git:(main) ✗ ./manage.py test --parallel
Found 2 test(s).
System check identified no issues (0 silenced).
..
----------------------------------------------------------------------
Ran 2 tests in 0.005s

OK

comment:9 by Matt Hegarty, 12 months ago

one final thought, the 'tutorial' test app passes when using python3.10 on Ubuntu (any version of django) - it might be worth you confirming that is the same for you.

comment:10 by Natalia Bidart, 12 months ago

Cc: Carlton Gibson Mariusz Felisiak added

I can confirm that the ./manage.py test --parallel works with any version of Django (4.0, 4.2, 5.0b1) and Python 3.10.

root@568680bc51ec:/tutorial# python --version
Python 3.10.13
root@568680bc51ec:/tutorial# python manage.py test --parallel
Found 2 test(s).
System check identified no issues (0 silenced).
..
----------------------------------------------------------------------
Ran 2 tests in 0.025s

OK

Given that this is an issue related to the Python version, I'm not sure how to proceed. Mariusz, Carlton, what would be your advice in this case?

comment:11 by Carlton Gibson, 12 months ago

So the tests don't run in parallel on macOS prior to Django 4.1 (release notes) so the whole pickle step never gets hit. I'd guess that would explain the issue there.

It seems that IsolatedAsyncioTestCase hasn't been (ever?) supported?

That's the nub of it.

Django's SimpleTestCase added equivalent support for async def test methods in Django 3.1 (docs), but IsolatedAsyncioTestCase wasn't part of that.

This ticket is maybe a feature request to add support for IsolatedAsyncioTestCase subclasses but what's the need, and given the error…

TypeError: cannot pickle '_contextvars.Context' object

… is it going to be feasible at all?

(It might be but, for me, I'd like some indication of a yes there before accepting such, so we don't send folks down a dead-end rabbit hole.)

comment:12 by Matt Hegarty, 12 months ago

Just to clarify, the tests pass with any version of Django if you use python3.10 (tested on MacOS and Ubuntu). They fail when running 3.11 or 3.12.

in reply to:  12 comment:13 by Carlton Gibson, 12 months ago

Replying to Matt Hegarty:

Just to clarify, the tests pass with any version of Django if you use python3.10 (tested on MacOS and Ubuntu). They fail when running 3.11 or 3.12.

I guess the question then is what change led to this in Python 3.11?

The addition of IsolatedAsyncioTestCase.enterAsyncContext() is a likely candidate.

But, still, use of IsolatedAsyncioTestCase has never been supported by Django, or tested in its test suite, so any success prior to 3.11 would be coincidental (leveraging undefined behaviour from Django's point of view). Adding such support would still be a new feature.

Last edited 12 months ago by Carlton Gibson (previous) (diff)

comment:14 by Matt Hegarty, 12 months ago

OK. Thanks all for investigating.

comment:15 by Natalia Bidart, 12 months ago

Resolution: needsinfowontfix

Setting to wontfix since that resolution aligns better with the latest outcome. Thanks everyone!

comment:16 by Natalia Bidart, 12 months ago

Summary: Parallel tests fail on Python 3.11+ and django 4.1.0+ when IsolatedAsyncioTestCase is usedParallel tests fail on Python 3.11+ when IsolatedAsyncioTestCase is used
Note: See TracTickets for help on using tickets.
Back to Top