runtests.py cleanup exception on Python 3.6

[Vytis Banaitis]

Per ​Tim's suggestion, I've looked into a cleanup exception that happens when running tests on Python 3.6:

$ ./tests/runtests.py basic
Testing against Django installed in '/home/tim/code/django/django' with up to 3 processes
....
Destroying test database for alias 'other'...
 Traceback (most recent call last):
  File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 254, in _run_finalizers
    finalizer()
  File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 186, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line 465, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line 463, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/django_k0xziymh/pymp-i4s112bj'

What I found out:

• multiprocessing is involved. Indeed, running with --parallel=1 does not trigger this error.
• Jinja2 is involved. Running tests without Jinja2 installed does not trigger this error.

What happens:

1. multiprocessing registers atexit handler.
2. runtests.py creates a temp dir (e.g. /tmp/django_k0xziymh) and registers atexit handler to remove it.
3. multiprocessing creates a temp dir (e.g. /tmp/django_k0xziymh/pymp-i4s112bj) which will be deleted in the handler it registered earlier.
4. Tests happen.
5. runtests.py exit handler deletes the temp dir.
6. multiprocessing exit handler tries to delete the inner temp dir but it is already gone.

On earlier Python versions 1 and 2 are swapped and 5 and 6 are swapped, so the error does not happen.

Jinja2 is imported by a chain of imports starting with from django.test import TestCase, TransactionTestCase (full chain below).
On Python 3.6 Jinja2 ​patches async support which eventually imports multiprocessing.util which registers the exit handler.

Possible solutions:

• Create the temp dir before importing django modules.
• Move some imports in runtests.py into functions, thereby delaying the indirect import of Jinja2.
• Move some imports somewhere else into functions to break the import chain.
• ...

The import chain leading up to the import of multiprocessing.util (with uninteresting import machinery stack frames removed):

  File "./runtests.py", line 18, in <module>
    from django.test import TestCase, TransactionTestCase
  File "/home/vytis/src/django/django/test/__init__.py", line 5, in <module>
    from django.test.client import Client, RequestFactory
  File "/home/vytis/src/django/django/test/client.py", line 12, in <module>
    from django.core.handlers.base import BaseHandler
  File "/home/vytis/src/django/django/core/handlers/base.py", line 7, in <module>
    from django.urls import get_resolver, set_urlconf
  File "/home/vytis/src/django/django/urls/__init__.py", line 1, in <module>
    from .base import (
  File "/home/vytis/src/django/django/urls/base.py", line 8, in <module>
    from .exceptions import NoReverseMatch, Resolver404
  File "/home/vytis/src/django/django/urls/exceptions.py", line 1, in <module>
    from django.http import Http404
  File "/home/vytis/src/django/django/http/__init__.py", line 5, in <module>
    from django.http.response import (
  File "/home/vytis/src/django/django/http/response.py", line 13, in <module>
    from django.core.serializers.json import DjangoJSONEncoder
  File "/home/vytis/src/django/django/core/serializers/__init__.py", line 23, in <module>
    from django.core.serializers.base import SerializerDoesNotExist
  File "/home/vytis/src/django/django/core/serializers/base.py", line 6, in <module>
    from django.db import models
  File "/home/vytis/src/django/django/db/models/__init__.py", line 3, in <module>
    from django.db.models.aggregates import *  # NOQA
  File "/home/vytis/src/django/django/db/models/aggregates.py", line 5, in <module>
    from django.db.models.expressions import Func, Star
  File "/home/vytis/src/django/django/db/models/expressions.py", line 6, in <module>
    from django.db.models import fields
  File "/home/vytis/src/django/django/db/models/fields/__init__.py", line 11, in <module>
    from django import forms
  File "/home/vytis/src/django/django/forms/__init__.py", line 6, in <module>
    from django.forms.boundfield import *  # NOQA
  File "/home/vytis/src/django/django/forms/boundfield.py", line 5, in <module>
    from django.forms.widgets import Textarea, TextInput
  File "/home/vytis/src/django/django/forms/widgets.py", line 21, in <module>
    from .renderers import get_default_renderer
  File "/home/vytis/src/django/django/forms/renderers.py", line 11, in <module>
    from django.template.backends.jinja2 import Jinja2
  File "/home/vytis/src/django/django/template/backends/jinja2.py", line 1, in <module>
    import jinja2
  File "/home/vytis/src/env/django-py36/lib/python3.6/site-packages/jinja2/__init__.py", line 81, in <module>
    _patch_async()
  File "/home/vytis/src/env/django-py36/lib/python3.6/site-packages/jinja2/__init__.py", line 77, in _patch_async
    from jinja2.asyncsupport import patch_all
  File "/home/vytis/src/env/django-py36/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 13, in <module>
    import asyncio
  File "/opt/python/lib/python3.6/asyncio/__init__.py", line 21, in <module>
    from .base_events import *
  File "/opt/python/lib/python3.6/asyncio/base_events.py", line 17, in <module>
    import concurrent.futures
  File "/opt/python/lib/python3.6/concurrent/futures/__init__.py", line 17, in <module>
    from concurrent.futures.process import ProcessPoolExecutor
  File "/opt/python/lib/python3.6/concurrent/futures/process.py", line 55, in <module>
    from multiprocessing.connection import wait
  File "/opt/python/lib/python3.6/multiprocessing/connection.py", line 23, in <module>
    from . import util

[Claude Paroz]

Another option might be to register a custom function that wraps shutil.rmtree and catches the exception.

[Vytis Banaitis]

Replying to Claude Paroz:

Another option might be to register a custom function that wraps shutil.rmtree and catches the exception.

It's possible, but would require monkey-patching multiprocessing.

[Claude Paroz]

Replying to Vytis Banaitis:

Replying to Claude Paroz:

Another option might be to register a custom function that wraps shutil.rmtree and catches the exception.

It's possible, but would require monkey-patching multiprocessing.

Really? My idea was something like:

def custom_delete(tmpdir):
    try:
        shutil.rmtree(tmpdir)
    except FileNotFoundError:
         pass

atexit.register(custom_delete, TMPDIR)

[Vytis Banaitis]

Replying to Claude Paroz:

Really? My idea was something like:

def custom_delete(tmpdir):
    try:
        shutil.rmtree(tmpdir)
    except FileNotFoundError:
         pass

atexit.register(custom_delete, TMPDIR)

Removal of TMPDIR happens without error.
The error is raised by multiprocessing exit handler which tries to delete a directory inside of TMPDIR.

[Claude Paroz]

Oh, now I see, sorry for the misunderstanding.

[Tim Graham]

I looked a little and didn't see an obvious solution besides some import rearranging as suggested in the ticket description.

[Tim Graham]

​PR. Not sure if it's ideal, but good enough for now?

[Tim Graham]

Rather than the rearranging imports approach of the first PR, an alternative ​PR deletes multiprocessing's temporary directory removal handler to avoid the error.

Going forward, patching cpython to ignore the error might be an option:

Lib/multiprocessing/util.py

@@ -113 +113 @@
         import shutil, tempfile
         tempdir = tempfile.mkdtemp(prefix='pymp-')
         info('created temp directory %s', tempdir)
-        Finalize(None, shutil.rmtree, args=[tempdir], exitpriority=-100)
+        Finalize(None, shutil.rmtree, args=[tempdir], kwargs={'ignore_errors': True}, exitpriority=-100)
         process.current_process()._config['tempdir'] = tempdir
     return tempdir
 

[Tim Graham <timograham@…>]

In 0c6c859:

Fixed #27890 -- Fixed FileNotFoundError cleanup exception in runtests.py on Python 3.6+.

[Tim Graham <timograham@…>]

In e0ddfa3a:

[1.11.x] Fixed #27890 -- Fixed FileNotFoundError cleanup exception in runtests.py on Python 3.6+.

Backport of 0c6c859d4edf6e462d000da21b4fa5009cb2696f from master