ImportError: partially initialized module '...' has no attribute '...'

[Collin Anderson]

Today I've been getting some occasional "ImportError"s when reloading the runserver. I have a feeling it's a race condition when importing my custom auth backend when runserver is reloading, likely caused by #33099 cached imports, committed on Friday. It doesn't happen every time which makes it harder to track down.

Maybe the cached_import logic needs to somehow check for a "partially initialized module" and use the slow-path in that case?

I'm not 100% it's not just an issue on my end, but I at least wanted to raise this here in case other people have issues and want to diagnose further.

Here's an example (ImportError when importing my custom auth backend):

Request Method: GET
Request URL: /cart/dropdown/

Django Version: 4.0.dev20210913065016
Python Version: 3.8.10

Template error:
In template templates/order/cart_dropdown.html, error at line 3
   Module "authbackend" does not define a "Bknd" attribute/class
   3 :  {% if request.order.ops %}


Traceback (most recent call last):
  File "django/template/base.py", line 862, in _resolve_lookup
    current = current[bit]

During handling of the above exception ('WSGIRequest' object is not subscriptable), another exception occurred:
  File "django/utils/module_loading.py", line 26, in import_string
    return cached_import(module_path, class_name)
  File "django/utils/module_loading.py", line 12, in cached_import
    return getattr(modules[module_path], class_name)

The above exception (partially initialized module 'authbackend' has no attribute 'Bknd' (most likely due to a circular import)) was the direct cause of the following exception:
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/shortcuts.py", line 19, in render
    content = loader.render_to_string(template_name, context, request, using=using)
  File "django/template/loader.py", line 62, in render_to_string
    return template.render(context, request)
  File "django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "django/template/base.py", line 176, in render
    return self._render(context)
  File "django/template/base.py", line 168, in _render
    return self.nodelist.render(context)
  File "django/template/base.py", line 977, in render
    return SafeString(''.join([
  File "django/template/base.py", line 978, in <listcomp>
    node.render_annotated(context) for node in self
  File "django/template/base.py", line 938, in render_annotated
    return self.render(context)
  File "django/template/defaulttags.py", line 386, in render
    return strip_spaces_between_tags(self.nodelist.render(context).strip())
  File "django/template/base.py", line 977, in render
    return SafeString(''.join([
  File "django/template/base.py", line 978, in <listcomp>
    node.render_annotated(context) for node in self
  File "django/template/base.py", line 938, in render_annotated
    return self.render(context)
  File "django/template/defaulttags.py", line 288, in render
    match = condition.eval(context)
  File "django/template/defaulttags.py", line 829, in eval
    return self.value.resolve(context, ignore_failures=True)
  File "django/template/base.py", line 701, in resolve
    obj = self.var.resolve(context)
  File "django/template/base.py", line 829, in resolve
    value = self._resolve_lookup(context)
  File "django/template/base.py", line 870, in _resolve_lookup
    current = getattr(current, bit)
  File "order/middleware.py", line 20, in __get__
    if not request.user.pk:  # if not logged in
  File "django/utils/functional.py", line 248, in inner
    self._setup()
  File "django/utils/functional.py", line 384, in _setup
    self._wrapped = self._setupfunc()
  File "django/contrib/auth/middleware.py", line 25, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "django/contrib/auth/__init__.py", line 183, in get_user
    backend = load_backend(backend_path)
  File "django/contrib/auth/__init__.py", line 21, in load_backend
    return import_string(path)()
  File "django/utils/module_loading.py", line 28, in import_string
    raise ImportError('Module "%s" does not define a "%s" attribute/class' % (

Exception Type: ImportError at /cart/dropdown/
Exception Value: Module "authbackend" does not define a "Bknd" attribute/class

[Keryn Knight]

Seems like a legitimate problem, that either requires working around (fastpath vs slowpath as mentioned) or reverting the change made in #33099.

It'd be interesting to see the middlewares & modules in play, even if they're reduced for privacy. My guess is circular imports are occurring possibly, or dynamic class creation?

[Mariusz Felisiak]

Regression in ecf87ad513fd8af6e4a6093ed918723a7d88d5ca.

[Mariusz Felisiak]

The easiest (but hacky) way to fix it is to check Python's internals and reload the module again:

diff --git a/django/utils/module_loading.py b/django/utils/module_loading.py
index 1df82b1c32..39cb784f72 100644
--- a/django/utils/module_loading.py
+++ b/django/utils/module_loading.py
@@ -7,7 +7,11 @@ from importlib.util import find_spec as importlib_find
 
 def cached_import(module_path, class_name):
     modules = sys.modules
-    if module_path not in modules:
+    if module_path not in modules or (
+        # Module it not fully initialize.
+        getattr(modules[module_path], '__spec__', None) is not None and
+        getattr(modules[module_path].__spec__, '_initializing', False) is True
+    ):
         import_module(module_path)
     return getattr(modules[module_path], class_name)
 

[Mariusz Felisiak]

importlib now uses the same hook, see ​https://github.com/python/cpython/commit/03648a2a91f9f1091cd21bd4cd6ca092ddb25640.

[Keryn Knight]

Serendipitous find :) For historical reference, it looks like _initializing is set via importlib._bootstrap._load_unlocked:

# This must be done before putting the module in sys.modules
# (otherwise an optimization shortcut in import.c becomes
# wrong).
spec._initializing = True

Possibly worth noting however is importlib.util._module_to_load which has equivalent but different (at least in 3.9.5):

# This must be done before putting the module in sys.modules
# (otherwise an optimization shortcut in import.c becomes wrong)
module.__initializing__ = True

If we're going to try and keep the cached_import, perhaps we need to accommodate both, or at least investigate the __initializing__ variation?

FWIW, the investigation/patch proposed above does change the performance profile (it's 5.64 µs for me now) but is still world's better than repeatedly doing import_module (at least until Python 3.11 which might improve things via the linked bpo-43392)

[Mariusz Felisiak]

​PR

Possibly worth noting however is importlib.util._module_to_load which has equivalent but different (at least in 3.9.5):

As far as I'm aware we shouldn't reach _module_to_load() using import_module(). Also Python uses the same strategy (checking __spec__._initialized) in few places, e.g. ​import_ensure_initialized().

[Collin Anderson]

By the way, I figured out I can just put time.sleep(5) at the top-level of my custom auth backend to reproduce the issue the patch does fix the issue. Thanks!

[GitHub <noreply@…>]

In 6426c307:

Fixed #33107 -- Fixed import_string() crash on not fully initialized modules.

Regression in ecf87ad513fd8af6e4a6093ed918723a7d88d5ca.

Thanks Collin Anderson for the report.

[GitHub <noreply@…>]

In a3185a6:

Refs #33107 -- Optimized cached_import() helper.