id summary reporter owner description type status component version severity resolution keywords cc stage has_patch needs_docs needs_tests needs_better_patch easy ui_ux 27890 runtests.py cleanup exception on Python 3.6 Vytis Banaitis nobody "Per [https://github.com/django/django/pull/8127#issuecomment-283322945 Tim's suggestion], I've looked into a cleanup exception that happens when running tests on Python 3.6: {{{ $ ./tests/runtests.py basic Testing against Django installed in '/home/tim/code/django/django' with up to 3 processes .... Destroying test database for alias 'other'... Traceback (most recent call last): File ""/opt/python3.6.0/lib/python3.6/multiprocessing/util.py"", line 254, in _run_finalizers finalizer() File ""/opt/python3.6.0/lib/python3.6/multiprocessing/util.py"", line 186, in __call__ res = self._callback(*self._args, **self._kwargs) File ""/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py"", line 465, in rmtree onerror(os.lstat, path, sys.exc_info()) File ""/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py"", line 463, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/django_k0xziymh/pymp-i4s112bj' }}} What I found out: - `multiprocessing` is involved. Indeed, running with `--parallel=1` does not trigger this error. - Jinja2 is involved. Running tests without Jinja2 installed does not trigger this error. What happens: 1. `multiprocessing` registers `atexit` handler. 1. `runtests.py` creates a temp dir (e.g. `/tmp/django_k0xziymh`) and registers `atexit` handler to remove it. 1. `multiprocessing` creates a temp dir (e.g. `/tmp/django_k0xziymh/pymp-i4s112bj`) which will be deleted in the handler it registered earlier. 1. Tests happen. 1. `runtests.py` exit handler deletes the temp dir. 1. `multiprocessing` exit handler tries to delete the inner temp dir but it is already gone. On earlier Python versions 1 and 2 are swapped and 5 and 6 are swapped, so the error does not happen. Jinja2 is imported by a chain of imports starting with `from django.test import TestCase, TransactionTestCase` (full chain below). On Python 3.6 Jinja2 [https://github.com/pallets/jinja/blob/master/jinja2/__init__.py#L74-L78 patches async support] which eventually imports `multiprocessing.util` which registers the exit handler. Possible solutions: - Create the temp dir before importing `django` modules. - Move some imports in `runtests.py` into functions, thereby delaying the indirect import of Jinja2. - Move some imports somewhere else into functions to break the import chain. - ... The import chain leading up to the import of `multiprocessing.util` (with uninteresting import machinery stack frames removed): {{{ File ""./runtests.py"", line 18, in from django.test import TestCase, TransactionTestCase File ""/home/vytis/src/django/django/test/__init__.py"", line 5, in from django.test.client import Client, RequestFactory File ""/home/vytis/src/django/django/test/client.py"", line 12, in from django.core.handlers.base import BaseHandler File ""/home/vytis/src/django/django/core/handlers/base.py"", line 7, in from django.urls import get_resolver, set_urlconf File ""/home/vytis/src/django/django/urls/__init__.py"", line 1, in from .base import ( File ""/home/vytis/src/django/django/urls/base.py"", line 8, in from .exceptions import NoReverseMatch, Resolver404 File ""/home/vytis/src/django/django/urls/exceptions.py"", line 1, in from django.http import Http404 File ""/home/vytis/src/django/django/http/__init__.py"", line 5, in from django.http.response import ( File ""/home/vytis/src/django/django/http/response.py"", line 13, in from django.core.serializers.json import DjangoJSONEncoder File ""/home/vytis/src/django/django/core/serializers/__init__.py"", line 23, in from django.core.serializers.base import SerializerDoesNotExist File ""/home/vytis/src/django/django/core/serializers/base.py"", line 6, in from django.db import models File ""/home/vytis/src/django/django/db/models/__init__.py"", line 3, in from django.db.models.aggregates import * # NOQA File ""/home/vytis/src/django/django/db/models/aggregates.py"", line 5, in from django.db.models.expressions import Func, Star File ""/home/vytis/src/django/django/db/models/expressions.py"", line 6, in from django.db.models import fields File ""/home/vytis/src/django/django/db/models/fields/__init__.py"", line 11, in from django import forms File ""/home/vytis/src/django/django/forms/__init__.py"", line 6, in from django.forms.boundfield import * # NOQA File ""/home/vytis/src/django/django/forms/boundfield.py"", line 5, in from django.forms.widgets import Textarea, TextInput File ""/home/vytis/src/django/django/forms/widgets.py"", line 21, in from .renderers import get_default_renderer File ""/home/vytis/src/django/django/forms/renderers.py"", line 11, in from django.template.backends.jinja2 import Jinja2 File ""/home/vytis/src/django/django/template/backends/jinja2.py"", line 1, in import jinja2 File ""/home/vytis/src/env/django-py36/lib/python3.6/site-packages/jinja2/__init__.py"", line 81, in _patch_async() File ""/home/vytis/src/env/django-py36/lib/python3.6/site-packages/jinja2/__init__.py"", line 77, in _patch_async from jinja2.asyncsupport import patch_all File ""/home/vytis/src/env/django-py36/lib/python3.6/site-packages/jinja2/asyncsupport.py"", line 13, in import asyncio File ""/opt/python/lib/python3.6/asyncio/__init__.py"", line 21, in from .base_events import * File ""/opt/python/lib/python3.6/asyncio/base_events.py"", line 17, in import concurrent.futures File ""/opt/python/lib/python3.6/concurrent/futures/__init__.py"", line 17, in from concurrent.futures.process import ProcessPoolExecutor File ""/opt/python/lib/python3.6/concurrent/futures/process.py"", line 55, in from multiprocessing.connection import wait File ""/opt/python/lib/python3.6/multiprocessing/connection.py"", line 23, in from . import util }}}" Bug closed Core (Other) 1.11 Release blocker fixed Accepted 1 0 0 0 0 0