Bundled importlib some times won't work properly with python3.3

[Andrei Antoukh]

In one of my project some times (the most common case, run runserver, change code, runsever reloads) and in some requests is raised strange KeyError:

Traceback (most recent call last):
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/core/handlers/base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/rest_framework/viewsets.py", line 78, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/rest_framework/views.py", line 399, in dispatch
    response = self.handle_exception(exc)
  File "/home/niwi/devel/greenmine-back/greenmine/base/monkey.py", line 23, in handle_exception
    return super(APIView, self).handle_exception(exc)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/rest_framework/views.py", line 396, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/rest_framework/mixins.py", line 139, in partial_update
    return self.update(request, *args, **kwargs)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/db/transaction.py", line 338, in inner
    return func(*args, **kwargs)
  File "/home/niwi/devel/greenmine-back/greenmine/base/api.py", line 28, in update
    return super().update(*args, **kwargs)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/rest_framework/mixins.py", line 132, in update
    self.post_save(self.object, created=created)
  File "/home/niwi/devel/greenmine-back/greenmine/projects/userstories/api.py", line 90, in post_save
    super().post_save(obj, created)
  File "/home/niwi/devel/greenmine-back/greenmine/base/notifications/api.py", line 25, in post_save
    context["changed_fields_dict"] = obj.get_changed_fields_dict(self.request.DATA)
  File "/home/niwi/devel/greenmine-back/greenmine/base/notifications/models.py", line 77, in get_changed_fields_dict
    field_dict.update(self._get_changed_field(field_name, data_value))
  File "/home/niwi/devel/greenmine-back/greenmine/base/notifications/models.py", line 126, in _get_changed_field
    old_value = self._get_changed_field_old_value(field_name, data_value)
  File "/home/niwi/devel/greenmine-back/greenmine/base/notifications/models.py", line 119, in _get_changed_field_old_value
    return (self.last_version and self.last_version.field_dict.get(field_name, data_value) or None)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/reversion/models.py", line 177, in field_dict
    object_version = self.object_version
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/reversion/models.py", line 164, in object_version
    return list(serializers.deserialize(self.format, data, ignorenonexistent=True))[0]
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/core/serializers/__init__.py", line 132, in deserialize
    d = get_deserializer(format)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/core/serializers/__init__.py", line 111, in get_deserializer
    _load_serializers()
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/core/serializers/__init__.py", line 144, in _load_serializers
    register_serializer(format, BUILTIN_SERIALIZERS[format], serializers)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/core/serializers/__init__.py", line 69, in register_serializer
    module = importlib.import_module(serializer_module)
  File "/home/niwi/.envs/greenmine/lib/python3.3/site-packages/django/utils/importlib.py", line 36, in import_module
    return sys.modules[name]
KeyError: 'django.core.serializers.pyyaml'

Saying code of bundled importlib...

def import_module(name, package=None):
    """Import a module.

    The 'package' argument is required when performing a relative import. It
    specifies the package to use as the anchor point from which to resolve the
    relative import to an absolute import.

    """
    if name.startswith('.'):
        if not package:
            raise TypeError("relative imports require the 'package' argument")
        level = 0
        for character in name:
            if character != '.':
                break
            level += 1
        name = _resolve_name(name[level:], package, level)
    __import__(name)
    return sys.modules[name]

... seems __import__ is called with "django.core.serializers.pyyaml" but it does nothing (not raises ImportError, but it should do it) and sys.modules, obviously not contains "django.core.serializers.pyyaml".

(Note: I don't have pyyaml installed and I not use it on my project...)

I don't have deterministic way to reproduce this, because it hapens sometimes using python3.3 and django1.6(alpha->rc1).

I think that it happens because a python2.7 importlib isn't work properly with python3 and a simple solution is use python3 importlib if you running django with python3:

​https://github.com/niwibe/django/commit/2f73c41ed0e6e357b6710ae3b5bce4d7d55870cc

I know that importlib is now deprecated on master, but I think it should fixed also in 1.5, 1.6 and master.

Thanks.

[Claude Paroz]

I think this is a safe change and I'm ready to do the backporting job. However, I'd like to get fellow core devs approval before going forward.

[Aymeric Augustin]

I'm OK with that.

[Claude Paroz <claude@…>]

In 3351e94ffa37d3824a49b687125ee4a430b69bbe:

Fixed #21335 -- Use importlib from python3 when using python3

[Claude Paroz <claude@…>]

In e052ada0f6f21208d0f215d2f9184c4069ac1419:

[1.6.x] Fixed #21335 -- Use importlib from python3 when using python3

Backport of 3351e94ffa from master.

[Claude Paroz <claude@…>]

In 1ef737416ab56a6becee36876769909a37541465:

[1.5.x] Fixed #21335 -- Use importlib from python3 when using python3

Backport of 3351e94ffa from master.

[Claude Paroz <claude@…>]

In ec0a3f5085219cd647d3e93841018987ce5384bf:

[1.6.x] Removed relative usage of import_module

Python 3 version of importlib doesn't support this syntax.
Partial backport of fdd7a355bf. Refs #21335.

[Claude Paroz <claude@…>]

In 3b4b376bd1dde3cbe64cf8f1a4fb1756904af710:

Revert "[1.5.x] Fixed #21335 -- Use importlib from python3 when using python3"

This reverts commit 1ef737416ab56a6becee36876769909a37541465.
This would require changing some import_module calls (relative
syntax, see ec0a3f50852), too much for a stable release.